maximo/us-dop-scrapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: maximo:v0.0.1
Branches Tags
maximo:master
maximo:v0.0.6
maximo:v0.0.5
maximo:v0.0.4
maximo:v0.0.3
maximo:v0.0.2
maximo:v0.0.1
...
compare: maximo:master
Branches Tags
maximo:master
maximo:v0.0.6
maximo:v0.0.5
maximo:v0.0.4
maximo:v0.0.3
maximo:v0.0.2
maximo:v0.0.1

11 Commits
v0.0.1 ... master

Author SHA1 Message Date
maximo tejeda
c0c55833aa tidy deps
All checks were successful
${{ github.actor }} executed Build Push Prod / build (push) Successful in 16m33s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Successful in 18s
Details
2025-03-07 22:16:46 -04:00
maximo tejeda
1551d04866 UPDATE pw
Some checks failed
${{ github.actor }} executed Build Push Prod / build (push) Failing after 1m40s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Has been skipped
Details
2025-03-07 22:10:17 -04:00
maximo tejeda
bb91d0c45d UPDATE PW version
Some checks failed
${{ github.actor }} executed Build Push Prod / build (push) Failing after 1m36s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Has been skipped
Details
2025-03-07 21:52:59 -04:00
maximo tejeda
0417c83f82 fix other typo
All checks were successful
dev test / test (push) Successful in 20s
Details
dev test / vulnCheck (push) Successful in 19s
Details
dev test / Ci-Lint (push) Successful in 1m1s
Details
${{ github.actor }} executed Build Push Prod / build (push) Successful in 16m30s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Successful in 16s
Details
2025-03-07 18:42:49 -04:00
maximo tejeda
fcf653109e fix typo
All checks were successful
dev test / test (push) Successful in 25s
Details
dev test / vulnCheck (push) Successful in 19s
Details
dev test / Ci-Lint (push) Successful in 1m11s
Details
${{ github.actor }} executed Build Push Prod / build (push) Successful in 16m31s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Successful in 17s
Details
2025-03-07 18:17:41 -04:00
maximo tejeda
f0ebc358f1 IGNORE err on delete kbuectl
Some checks failed
dev test / test (push) Successful in 19s
Details
dev test / vulnCheck (push) Successful in 18s
Details
dev test / Ci-Lint (push) Successful in 1m5s
Details
${{ github.actor }} executed Build Push Prod / build (push) Successful in 16m37s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Failing after 16s
Details
2025-03-07 17:55:46 -04:00
maximo tejeda
12bc8a68b8 unproxxied domain
Some checks failed
dev test / test (push) Successful in 1m49s
Details
dev test / vulnCheck (push) Successful in 1m15s
Details
dev test / Ci-Lint (push) Successful in 32s
Details
${{ github.actor }} executed Build Push Prod / build (push) Successful in 17m0s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Failing after 16s
Details
2025-03-07 17:18:37 -04:00
maximo tejeda
76e8f1f44a add local server
Some checks failed
dev test / test (push) Successful in 1m29s
Details
dev test / vulnCheck (push) Successful in 46s
Details
dev test / Ci-Lint (push) Successful in 33s
Details
${{ github.actor }} executed Build Push Prod / build (push) Failing after 12s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Has been skipped
Details
2025-03-07 17:10:34 -04:00
maximo tejeda
4dc90089bd IGNORE delte error before apply in case of not exists
Some checks failed
dev test / test (push) Failing after 2m49s
Details
dev test / vulnCheck (push) Has been skipped
Details
dev test / Ci-Lint (push) Has been skipped
Details
${{ github.actor }} executed Build Push Prod / build (push) Failing after 20m37s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Has been skipped
Details
2025-03-07 13:58:19 -04:00
maximo tejeda
9b6267204a Delete old deployment 2025-03-07 13:56:23 -04:00
maximo tejeda
14e54b060d new approach
Some checks failed
${{ github.actor }} executed Build Push Prod / build (push) Successful in 18m27s
Details
${{ github.actor }} executed Build Push Prod / deploy (push) Failing after 16s
Details
2024-12-04 11:26:57 -04:00
5 changed files with 27 additions and 283 deletions
Show Stats Expand all files Collapse all files

4
.github/workflows/build_push_and_deploy.yaml vendored
View File

@ -75,7 +75,7 @@ jobs:
BPD: "${{ vars.BPD }}"
GENERAL: "${{ vars.GENERAL }}"
SCOTIA: "${{ vars.SCOTIA }}"
VIMECA: "${{ vars.VIMENCA }}"
VIMENCA: "${{ vars.VIMENCA }}"
steps:
- name: echo repo dir
id: repo-url
@ -111,6 +111,6 @@ jobs:
export IMAGE="$( echo ${{ github.server_url }} | sed -e 's;https://;;')/${{ github.repository }}:${{ github.ref_name }}"
echo "${{ secrets.KUBECONFIG }}" > ~/.kube/config
- name: delete olf config
run: kubectl delete -f k8s/cronjobs.yml
run: kubectl delete -f k8s/cronjobs.yml || true
- name: apply config
run: kubectl apply -f k8s/cronjobs.yml

21
Dockerfile
View File

@ -1,16 +1,31 @@
# Stage 1: Modules caching
FROM golang:latest as modules
COPY go.mod go.sum /modules/
WORKDIR /modules
RUN go mod download
FROM golang:latest AS builder
ARG TARGETARCH
ARG version=not-set
ARG SHORTSHA=not-set
COPY --from=modules /go/pkg /go/pkg
RUN apt update && apt install git
WORKDIR /app
COPY . .
RUN go mod tidy
RUN go run github.com/playwright-community/playwright-go/cmd/playwright@latest install --with-deps
# Install playwright cli with right version for later use
RUN PWGO_VER=$(grep -oE "playwright-go v\S+" /app/go.mod | sed 's/playwright-go //g') \
&& go install github.com/playwright-community/playwright-go/cmd/playwright@${PWGO_VER}
# https://stackoverflow.com/questions/70369368/check-architecture-in-dockerfile-to-get-amd-arm
RUN go build -o bin/crawler \
-ldflags "-X main.Shortsha=${SHORTSHA} \
-X main.Version=${version} \
-X main.Aarch=${TARGETARCH}" ./cmd/crawler/main.go
ENTRYPOINT /app/bin/crawler
FROM ubuntu:noble
COPY --from=builder /go/bin/playwright /app/bin/crawler /
RUN apt-get update && apt-get install -y ca-certificates tzdata \
# Install dependencies and all browsers (or specify one)
&& /playwright install --with-deps \
&& rm -rf /var/lib/apt/lists/*
CMD ["/crawler"]

4
go.mod
View File

@ -7,20 +7,18 @@ toolchain go1.23.2
require (
git.maximotejeda.com/maximo/dolar v0.0.1
github.com/nats-io/nats.go v1.37.0
github.com/playwright-community/playwright-go v0.4802.0
github.com/playwright-community/playwright-go v0.5001.0
golang.org/x/text v0.20.0
google.golang.org/grpc v1.68.0
)
require (
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964 // indirect
github.com/deckarep/golang-set/v2 v2.6.0 // indirect
github.com/go-jose/go-jose/v3 v3.0.3 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/nats-io/nkeys v0.4.8 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.29.0 // indirect
golang.org/x/net v0.31.0 // indirect
golang.org/x/sys v0.27.0 // indirect

40
go.sum
View File

@ -1,77 +1,56 @@
git.maximotejeda.com/maximo/dolar v0.0.1 h1:pRO0qHRHzKlTsZtXRM4QT8VjPsS2AwOduOKIISWbXTU=
git.maximotejeda.com/maximo/dolar v0.0.1/go.mod h1:I3q+2GDNEwMSfSLg/7g5lkI+EWPR/90JvMO94KoFKBs=
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964 h1:y5HC9v93H5EPKqaS1UYVg1uYah5Xf51mBfIoWehClUQ=
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964/go.mod h1:Xd9hchkHSWYkEqJwUGisez3G1QY8Ryz0sdWrLPMGjLk=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM=
github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
github.com/go-jose/go-jose/v3 v3.0.1 h1:pWmKFVtt+Jl0vBZTIpz/eAKwsm6LkIxDVVbFHKkchhA=
github.com/go-jose/go-jose/v3 v3.0.1/go.mod h1:RNkWWRld676jZEYoV3+XK8L2ZnNSvIsxFMht0mSX+u8=
github.com/go-jose/go-jose/v3 v3.0.3 h1:fFKWeig/irsp7XD2zBxvnmA/XaRWp5V3CBsZXJF7G7k=
github.com/go-jose/go-jose/v3 v3.0.3/go.mod h1:5b+7YgP7ZICgJDBdfjZaIt+H/9L9T/YQrVfLAMboGkQ=
github.com/go-stack/stack v1.8.1 h1:ntEHSVwIt7PNXNpgPmVfMrNhLtgjlmnZha2kOpuRiDw=
github.com/go-stack/stack v1.8.1/go.mod h1:dcoOX6HbPZSZptuspn9bctJ+N/CnF5gGygcUP3XYfe4=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4=
github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/mitchellh/go-ps v1.0.0 h1:i6ampVEEF4wQFF+bkYfwYgY+F/uYJDktmvLPf7qIgjc=
github.com/mitchellh/go-ps v1.0.0/go.mod h1:J4lOc8z8yJs6vUwklHw2XEIiT4z4C40KtWVN3nvg8Pg=
github.com/nats-io/nats.go v1.34.1 h1:syWey5xaNHZgicYBemv0nohUPPmaLteiBEUT6Q5+F/4=
github.com/nats-io/nats.go v1.34.1/go.mod h1:Ubdu4Nh9exXdSz0RVWRFBbRfrbSxOYd26oF0wkWclB8=
github.com/nats-io/nats.go v1.37.0 h1:07rauXbVnnJvv1gfIyghFEo6lUcYRY0WXc3x7x0vUxE=
github.com/nats-io/nats.go v1.37.0/go.mod h1:Ubdu4Nh9exXdSz0RVWRFBbRfrbSxOYd26oF0wkWclB8=
github.com/nats-io/nkeys v0.4.7 h1:RwNJbbIdYCoClSDNY7QVKZlyb/wfT6ugvFCiKy6vDvI=
github.com/nats-io/nkeys v0.4.7/go.mod h1:kqXRgRDPlGy7nGaEDMuYzmiJCIAAWDK0IMBtDmGD0nc=
github.com/nats-io/nkeys v0.4.8 h1:+wee30071y3vCZAYRsnrmIPaOe47A/SkK/UBDPdIV70=
github.com/nats-io/nkeys v0.4.8/go.mod h1:kqXRgRDPlGy7nGaEDMuYzmiJCIAAWDK0IMBtDmGD0nc=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/playwright-community/playwright-go v0.4001.0 h1:2cBiTIjCvFu7zUrZ48C0YC2DIp90Tbudueq4brUGjHM=
github.com/playwright-community/playwright-go v0.4001.0/go.mod h1:quEkYFrvvpQyGSxBjnYbGS52vrUDB2uaY1cOzkkSHCc=
github.com/playwright-community/playwright-go v0.4802.0 h1:FSuvi5Pg/xp+n7vFpu2wGldwSQ3grsaDlHFRfHRQiy4=
github.com/playwright-community/playwright-go v0.4802.0/go.mod h1:kBNWs/w2aJ2ZUp1wEOOFLXgOqvppFngM5OS+qyhl+ZM=
github.com/playwright-community/playwright-go v0.5001.0 h1:EY3oB+rU9cUp6CLHguWE8VMZTwAg+83Yyb7dQqEmGLg=
github.com/playwright-community/playwright-go v0.5001.0/go.mod h1:kBNWs/w2aJ2ZUp1wEOOFLXgOqvppFngM5OS+qyhl+ZM=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30=
golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M=
golang.org/x/crypto v0.29.0 h1:L5SG1JTTXupVV3n6sUqMTeWbjAyfPwoda2DLX8J8FrQ=
golang.org/x/crypto v0.29.0/go.mod h1:+F4F4N5hv6v38hfeYwTdx20oUvLLc+QfrE9Ax9HtgRg=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.27.0 h1:5K3Njcw06/l2y9vpGCSdcxWOYHOUk3dVNGDXN+FvAys=
golang.org/x/net v0.27.0/go.mod h1:dDi0PyhWNoiUOrAS8uXv/vnScO4wnHQO4mj9fn/RytE=
golang.org/x/net v0.31.0 h1:68CPQngjLL0r2AlUKiSxtQFKvzRVbnzLwMUn5SzcLHo=
golang.org/x/net v0.31.0/go.mod h1:P4fl1q7dY2hnZFxEk4pPSkDHF+QqjitcnDjUQyMM+pM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@ -79,8 +58,6 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.27.0 h1:wBqf8DvsY9Y/2P8gAfPDEYNuS30J4lPHJxXSb/nJZ+s=
golang.org/x/sys v0.27.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@ -94,8 +71,6 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/text v0.20.0 h1:gK/Kv2otX8gz+wn7Rmb3vT96ZwuoxnQlY+HlJVj7Qug=
golang.org/x/text v0.20.0/go.mod h1:D4IsuqiFMhST5bX19pQ9ikHC2GsaKyk/oF+pn3ducp4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@ -103,17 +78,10 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240711142825-46eb208f015d h1:JU0iKnSg02Gmb5ZdV8nYsKEKsP6o/FGVWTrw4i1DA9A=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240711142825-46eb208f015d/go.mod h1:Ue6ibwXGpU+dqIcODieyLOcgj7z8+IcskoNIgZxtrFY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a h1:hgh8P4EuoxpsuKMXX/To36nOFD7vixReXgn8lPGnt+o=
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a/go.mod h1:5uTbfoYQed2U9p3KIj2/Zzm02PYhndfdmML0qC3q3FU=
google.golang.org/grpc v1.65.0 h1:bs/cUb4lp1G5iImFFd3u5ixQzweKizoZJAwBNLR42lc=
google.golang.org/grpc v1.65.0/go.mod h1:WgYC2ypjlB0EiQi6wdKixMqukr6lBc0Vo+oOgjrM5ZQ=
google.golang.org/grpc v1.68.0 h1:aHQeeJbo8zAkAa3pRzrVjZlbz6uSfeOXlJNQM0RAbz0=
google.golang.org/grpc v1.68.0/go.mod h1:fmSPC5AsjSBCK54MyHRx48kpOti1/jRfOlwEWywNjWA=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
google.golang.org/protobuf v1.35.2 h1:8Ar7bF+apOIoThw1EdZl0p1oWvMqTHmpA2fRTyZO8io=
google.golang.org/protobuf v1.35.2/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

237
k8s/cronjobs.yml
View File

@ -1,237 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: inf-cronjob
spec:
schedule: "*/20 8-19 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-inf
image: localhost:32000/crawler:latest
env:
- name: WHO
value: inf
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
- name: GENERAL
value: https://www.infodolar.com.do/
- name: NATSURI
value: "nats://nats-svc:4222"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: bcd-cronjob
spec:
schedule: "3,33 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-bcd
image: localhost:32000/crawler:latest
env:
- name: BCD
value: https://www.bancentral.gov.do/SectorExterno/HistoricoTasas
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: bcd
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: bpd-cronjob
spec:
schedule: "5,35 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-bpd
image: localhost:32000/crawler:latest
env:
- name: BPD
value: https://popularenlinea.com/empresarial/Paginas/Home.aspx
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: bpd
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: bhd-cronjob
spec:
schedule: "8,38 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-bhd
image: localhost:32000/crawler:latest
env:
- name: BHD
value: https://bhd.com.do/calculators?calculator=DIVISAS
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: bhd
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: brd-cronjob
spec:
schedule: "22,52 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-brd
image: localhost:32000/crawler:latest
env:
- name: BDR
value: https://www.banreservas.com/calculadoras
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: brd
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: apap-cronjob
spec:
schedule: "25,55 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-apap
image: localhost:32000/crawler:latest
env:
- name: APA
value: https://apap.com.do/
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: apap
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: bnc-cronjob
spec:
schedule: "11,41 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-bnc
image: localhost:32000/crawler:latest
env:
- name: BNC
value: https://www.banesco.com.do/
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: bnc
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: vimenca-cronjob
spec:
schedule: "14,44 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-vimenca
image: localhost:32000/crawler:latest
env:
- name: VIMENCA
value: https://www.bancovimenca.com/
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: vimenca
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: scotia-cronjob
spec:
schedule: "17,47 8-22 * * 1-6"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: crawler-scotia
image: localhost:32000/crawler:latest
env:
- name: SCOTIA
value: https://do.scotiabank.com/banca-personal/tarifas/tasas-de-cambio.html
- name: NATSURI
value: "nats://nats-svc:4222"
- name: WHO
value: scotia
- name: DOLLAR_SERVICE_URL
value: "dolar-grpc-svc:80"