Feat / Kujira

[danilo-silva-funttastic]

Summary
The Funttastic Labs and Robotter.AI team requests the inclusion of the Kujira DEX connector to the Hummingbot Gateway and Client. Kujira will enhance Humminbot features and capabilities and will be able to use any of the Hummingbot strategies and/or the customized scripts.

PRP: https://snapshot.org/#/hbot-prp.eth/proposal/0x7dbd4a6f3cc7460ca6f56415a57b9727ca7a9227be625efdc4e71dee3d0d0781
Tags: DEX, Gateway, Client, Connector, Kujira
Github handles: https://github.com/funttastic and https://github.com/yourtrading-ai
Kujira Blockchain Website: https://kujira.app/

Motivation

• High-Quality Code and Robust Testing Coverage:
• The Kujira connector will boast high-quality code implementation with a well-designed API and a robust testing coverage.
• Also, our API also counts with more than the common endpoints but with a great addition of other ones where you can to retrieve one, several or all (when applicable) markets, order books, tickers, and transactions at once; also create, cancel, and get one, several, and all (when applicable) orders at once, with buy or sell sides, of types market or limit, and so on.
• Our test coverage counts with unit and the integration tests in a full flow manner, where we developed an entire flow covering all the users operations, guaranteeing in this way that our users are always covered, not matter the changes.
• Opening the Door to Further Connectors: The Kujira connector, using the CLOB SPOT connector standard implemented in Injective, will pave the way for the creation of additional connectors in the future, thus enhancing the flexibility and capabilities of Hummingbot.
• Kujira Key Points:
• Sustainable Sovereign Layer 1 Ecosystem
• Focuses on sustainable FinTech solutions for protocols, builders, and web3 users
• Offers a range of innovative and user-friendly products that generate revenue and aim to transform payment infrastructure
• Provides easy-to-use, cost-effective tools for people of all levels of crypto experience
• Inclusive of all blockchains and networks, and offers products that do not require participants to risk more than they can afford to lose
• Aims to allow retail investors to access opportunities previously only available to the wealthy and elite
• Mission is to help people generate income regardless of market direction
• Semi-Permissioned:
• On-chain Scheduler:
• Native Token Generation:
• Build with Cosmos SDK:
• CosmWASM 1.0 Support:
• Thriving Community:
• Built to Last
• Kujira Innovative and User-friendly tools: Kujira's decentralized ecosystem offers a range of innovative, user-friendly tools and products. Some of they are:
• ORCA: Liquidate multiple assets across the whole of Cosmos at the click of a button. No bots, no code, no problems.
• FIN: Cosmos' first decentralized, permissionless, 100% on-chain order book style token exchange.
• BOW: BOW delivers a better trading experience on FIN by narrowing the price spread deepening the pool.
• GHOST: Mobilize your assets by lending them out to earn interest and secure working liquidity by using them as collateral.
• PILOT: Launch tokens in a fair and equitable manner that benefits and gives equal access to all.), BLUE (Your KUJI Dashboard. Stake KUJI, Vote on Governance, Swap KUJI, Bridge to Kujira Network, and claim rewards; all in one place.
• FINDER: A tool to search for present and historical data on the Kujira blockchain.

[danilo-silva-funttastic]

@fengtality @cardosofede @nikspz, I'm referring you here so you can check that we have now opened this PR here and also one for the client (hummingbot repo).

[fengtality]

@danilo-silva-funttastic Note that the chain routes are being refactored:

#147
hummingbot/hummingbot#6445

Please start incorporating these changes into this PR

[danilo-silva-funttastic]

@fengtality, @nikspz

We have reviewed all of the changes and removed all of the "extra" code from the PR.
We have also tested our code, and unit tests.
Our coverage is now in 81% for our connector.
We might be ready for merging.
We kindly ask you all to review it and all being ok approve our merging.

[fengtality]

See comment above

[fengtality]

I don't agree that we should add these Kujira specific routes. The point of #147 was to standardize common interactions with various chains under a single set of routes.

These routes below introduce different patterns for interactions such as fetching tokens that are already handled by the standard routes. This Kujira connector should fit into the existing model, and changes to the model should be proposed via PRP.

[danilo-silva-funttastic]

We will make changes to do not expose the specific routes.

[danilo-silva-funttastic]

We only make one request though, our current test coverage is based on that and we have 81% coverage now.
Because we have much more functionality (and we would like to maintain a separate fork then) we would need to do a too much big refactor if we try to remove everything that is apart from the current CLOB routes.
So what we propose is basically change the code so no one from the Client and the Gateway would be able to use the specific Kujira routes (and we will remove all swagger, json responses, and other related things from here) but if they want they could then use our fork which would provide some extra routes. Would that be ok @fengtality?

[rapcmia]

PR update

• Setup this PR and PR6399
• Successfully connected gateway:
  • gateway list ✅
  • gateway connect ✅
• Setup wallet using kepler and able to connect using mnemonic passphrase for both mainnet and testnet
• Check gateway config kujira and nodeURL is set to none
  • Check kujira.yml and set to null # Empty or null means choosing the RPC automatically with several nodes for testnet and mainnet
• When checking balance (even no balance), we are experiencing this error on testnet which seems to be related to incorrect mnemonic phrase but its the same wallet when calling the gateway connect, is there a diffenerent wallet to use for testnet?:
  
  
  2023-07-10 17:57:52,115 - 23736 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/wallet/add failed. See logs for more details. 2023-07-10 17:57:52,115 - 23736 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Invalid Kujira mnemonic.', 'httpErrorCode': 404, 'errorCode': -1, 'stack': 'Error: Invalid Kujira mnemonic.\n at throwIfErrorsExist (/Users/rapcomia/github/gateway/138/dist/src/services/validators.js:38:15)\n at /Users/rapcomia/github/gateway/138/dist/src/services/validators.js:103:40\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:25:58\n at Generator.next (<anonymous>)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:8:71\n at new Promise (<anonymous>)\n at __awaiter (/Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:4:12)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:24:86\n at /Users/rapcomia/github/gateway/138/dist/src/services/error-handler.js:37:28\n at Layer.handle [as handle_request] (/Users/rapcomia/github/gateway/138/node_modules/express/lib/router/layer.js:95:5)'} Traceback (most recent call last): File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper return await c File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/command/gateway_command.py", line 256, in _gateway_connect wallet_address, additional_prompt_values = await self._prompt_for_wallet_address( File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/command/gateway_command.py", line 382, in _prompt_for_wallet_address response: Dict[str, Any] = await self._get_gateway_instance().add_wallet( File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 266, in add_wallet return await self.api_request(method="post", path_url="wallet/add", params=request) File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 219, in api_request raise e File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 207, in api_request raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}") ValueError: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Invalid Kujira mnemonic.', 'httpErrorCode': 404, 'errorCode': -1, 'stack': 'Error: Invalid Kujira mnemonic.\n at throwIfErrorsExist (/Users/rapcomia/github/gateway/138/dist/src/services/validators.js:38:15)\n at /Users/rapcomia/github/gateway/138/dist/src/services/validators.js:103:40\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:25:58\n at Generator.next (<anonymous>)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:8:71\n at new Promise (<anonymous>)\n at __awaiter (/Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:4:12)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:24:86\n at /Users/rapcomia/github/gateway/138/dist/src/services/error-handler.js:37:28\n at Layer.handle [as handle_request] (/Users/rapcomia/github/gateway/138/node_modules/express/lib/router/layer.js:95:5)'}
  

---

Hi @danilo-silva-funttastic good day

• Can you advise where can we setup our wallet for testnet and if there is faucet available?
• How can we make use of the devnet?

[danilo-silva-funttastic]

PR update

• Setup this PR and PR6399

• Successfully connected gateway:

  • gateway list ✅
  • gateway connect ✅

• Setup wallet using kepler and able to connect using mnemonic passphrase for both mainnet and testnet

• Check gateway config kujira and nodeURL is set to none

  • Check kujira.yml and set to null # Empty or null means choosing the RPC automatically with several nodes for testnet and mainnet

• When checking balance (even no balance), we are experiencing this error on testnet which seems to be related to incorrect mnemonic phrase but its the same wallet when calling the gateway connect, is there a diffenerent wallet to use for testnet?:
  
  
  2023-07-10 17:57:52,115 - 23736 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/wallet/add failed. See logs for more details. 2023-07-10 17:57:52,115 - 23736 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Invalid Kujira mnemonic.', 'httpErrorCode': 404, 'errorCode': -1, 'stack': 'Error: Invalid Kujira mnemonic.\n at throwIfErrorsExist (/Users/rapcomia/github/gateway/138/dist/src/services/validators.js:38:15)\n at /Users/rapcomia/github/gateway/138/dist/src/services/validators.js:103:40\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:25:58\n at Generator.next (<anonymous>)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:8:71\n at new Promise (<anonymous>)\n at __awaiter (/Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:4:12)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:24:86\n at /Users/rapcomia/github/gateway/138/dist/src/services/error-handler.js:37:28\n at Layer.handle [as handle_request] (/Users/rapcomia/github/gateway/138/node_modules/express/lib/router/layer.js:95:5)'} Traceback (most recent call last): File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper return await c File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/command/gateway_command.py", line 256, in _gateway_connect wallet_address, additional_prompt_values = await self._prompt_for_wallet_address( File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/command/gateway_command.py", line 382, in _prompt_for_wallet_address response: Dict[str, Any] = await self._get_gateway_instance().add_wallet( File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 266, in add_wallet return await self.api_request(method="post", path_url="wallet/add", params=request) File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 219, in api_request raise e File "/Users/rapcomia/github/hummingbot/6399/hummingbot/core/gateway/gateway_http_client.py", line 207, in api_request raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}") ValueError: Error on POST https://localhost:15888/wallet/add Error: {'message': 'Invalid Kujira mnemonic.', 'httpErrorCode': 404, 'errorCode': -1, 'stack': 'Error: Invalid Kujira mnemonic.\n at throwIfErrorsExist (/Users/rapcomia/github/gateway/138/dist/src/services/validators.js:38:15)\n at /Users/rapcomia/github/gateway/138/dist/src/services/validators.js:103:40\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:25:58\n at Generator.next (<anonymous>)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:8:71\n at new Promise (<anonymous>)\n at __awaiter (/Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:4:12)\n at /Users/rapcomia/github/gateway/138/dist/src/services/wallet/wallet.routes.js:24:86\n at /Users/rapcomia/github/gateway/138/dist/src/services/error-handler.js:37:28\n at Layer.handle [as handle_request] (/Users/rapcomia/github/gateway/138/node_modules/express/lib/router/layer.js:95:5)'}
  

Hi @danilo-silva-funttastic good day

• Can you advise where can we setup our wallet for testnet and if there is faucet available?
• How can we make use of the devnet?

Hello @rapcmia , thanks for getting back to us ;)

Because of some new changes asked, we may need some time to conform to the changes.

I may drop a message here when we are ready for a new test ;)

In any case, when adding a Kujira wallet, from Kepler, the private key would be the full mnemonic phrase.

The testnet RPC that it seems to be the most stable these days would be this one here (so maybe it would be better to test with it instead of using the automatic option): https://kujira-testnet-rpc.polkachu.com

We have also made this very simple quickstart guide for the people that wants to get started: https://github.com/funttastic/kujira-quickstart-guide

Either way, answering the question about the faucet, yes, Kujira has one, here are the instructions:

Adding funds to a Testnet Wallet (optional)
In order to add funds to your wallet, you can use a faucet inside the Kujira Discord.

To join their discord you can use this link:

https://discord.gg/teamkujira

After joining and doing their verification process, you can look for this channel:

#public-testnet-faucet

Or try this link:

https://discord.com/channels/970650215801569330/1009931570263629854

Then you can use the following command there:

!faucet <change to your kujira wallet address here>

After that you should receive some Kujira tokens on your balance.

If you need more you can contact us here:

https://www.funttastic.com/discord

Thanks for pointing out the error with the balance.
We will double check it here ;)

Thanks a lot for your time, we appreciate it a lot!

[danilo-silva-funttastic]

@fengtality, @nikspz , @rapcmia

We have refactored our code and we tried to make the changes to be as simple as possible for the PR.

Please kindly review when you have time.

[danilo-silva-funttastic]

@rapcmia, in order to easy the testing, here are some tips from our side ;)

Please use Keplr for creating a Kujira wallet, the mnemonic is the private key that needs to be informed in the Client

To add funds to your wallet you can join to the Kuijra Discord (https://discord.gg/teamkujira),
pass their verification process, go to the #public-testnet-faucet channel (or use this link https://discord.com/channels/970650215801569330/1009931570263629854) and the use:

!faucet <change to your kujira wallet address here>

Preferably use this testnet RPC:
https://kujira-testnet-rpc.polkachu.com/

We have checked the balances, also for an empty wallet, and the PMM strategy. From our side everything is now passing.

Please note that because of some limitations of the clob endpoints, we needed to relax one validation there to enable the possibility to retrieve or cancel all orders if you don't inform a order id there.

If you have any doubts please let us know (you can also contact me directly there on Discord if you prefer).

Thanks in advance,
Danilo.

[rapcmia]

PR update:

• Test latest commit cd64110 with Feat / Kujira hummingbot#6399
• Successfully setup gateway ✅
• Successfully connected wallet using kepler on testnet
  • Balance displayed ok
• Checking gateway config kujira
  • We would want to confirm what is the difference between gateway config kujira.network and kujira.networks?
    

---

Setup on https://fin.kujira.network/ using KUJI/USK and KUJI/DEMO using Pure Market-making:

• Able to start PMM with KUJI/DEMO, observed:
  • Orders are successfully created and canceled
  • Cancelation would have delays of 10s and sometimes orders get stuck and not recognized by the client when start again (test in progress)
  • However for the logs panel does not seem to be displaying correctly
    
  • Setup tight spreads for order filled events
    
    • ⚠️ The number of trades from client is half the value of Filled Orders from https://fin.kujira.network/
    • Also. we would like to know if we need to Claim all?

When starting the client, getting unhandled error in background for KUJI/USDK

2023-07-14 19:45:22,869 - 16657 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - cancel_all_orders: start
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - _update_markets: start
2023-07-14 19:45:22,871 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets request:
 "{
  "connector": "kujira",
  "chain": "kujira",
  "network": "testnet",
  "trading_pair": "KUJI-USDK"
}".
2023-07-14 19:45:22,919 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets response:
 "{
  "network": "testnet",
  "timestamp": 1689335122875,
  "latency": 0.001,
  "markets": {}
}".
2023-07-14 19:45:22,920 - 16657 - hummingbot.client.hummingbot_application - ERROR - Error canceling outstanding orders.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/hummingbot_application.py", line 256, in _cancel_outstanding_orders
    cancellation_results = await market.cancel_all(kill_timeout)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 723, in cancel_all
    return await self._api_data_source.cancel_all(timeout_seconds)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 789, in cancel_all
    return await self.cancel_all_orders()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 388, in cancel_all_orders
    self._check_markets_initialized() or await self._update_markets()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 723, in _update_markets
    self._market = self._markets[self._trading_pair]
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/dotmap/__init__.py", line 99, in __getitem__
    return self._map[k]
KeyError: 'KUJI-USDK'

Run stop command getting Error canceling outstanding orders and client is stuck, we have to run exit -f so we can start anew.

2023-07-14 19:45:22,869 - 16657 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - cancel_all_orders: start
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - _update_markets: start
2023-07-14 19:45:22,871 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets request:
 "{
  "connector": "kujira",
  "chain": "kujira",
  "network": "testnet",
  "trading_pair": "KUJI-USDK"
}".
2023-07-14 19:45:22,919 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets response:
 "{
  "network": "testnet",
  "timestamp": 1689335122875,
  "latency": 0.001,
  "markets": {}
}".
2023-07-14 19:45:22,920 - 16657 - hummingbot.client.hummingbot_application - ERROR - Error canceling outstanding orders.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/hummingbot_application.py", line 256, in _cancel_outstanding_orders
    cancellation_results = await market.cancel_all(kill_timeout)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 723, in cancel_all
    return await self._api_data_source.cancel_all(timeout_seconds)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 789, in cancel_all
    return await self.cancel_all_orders()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 388, in cancel_all_orders
    self._check_markets_initialized() or await self._update_markets()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 723, in _update_markets
    self._market = self._markets[self._trading_pair]
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/dotmap/__init__.py", line 99, in __getitem__
    return self._map[k]
KeyError: 'KUJI-USDK'

So for testnet, are we ok with using KUJI/USK for running tests? thank you 🙇🏼

Steps to reproduce:

• Connect to kepler wallet and use testnet
• Setup simple PMM for both KUJI\USK and KUJI\DEMO
• Observe connector behavior

logs_pmm-kujira.log

[danilo-silva-funttastic]

Hey @rapcmia , thanks for your valuable feedback and tests :)

Below I'll be commenting on some questions

PR update:

• Test latest commit cd64110 with Feat / Kujira hummingbot#6399

• Successfully setup gateway ✅

• Successfully connected wallet using kepler on testnet

  • Balance displayed ok

• Checking gateway config kujira

  • We would want to confirm what is the difference between gateway config kujira.network and kujira.networks?

The kujira.networks is like an enumerator listing all of the available networks, where the user can customize the nodeURL to be used. Also is mainly used to compound the availableNetworks of Kujira for the src/connectors/connectors.routes.ts file.

The kujira.network is the default network to be used in case the request does not inform one and internally for our unit tests and integration tests. But since all our call from the client inform a chain, network and a connector, this kuijria.network will not be used when the client interacts. We can probably remove this config or leave it there in the code only, not in the yml, we are going to check this here ;)

![image](https://user-images.githubusercontent.com/73840223/253570168-9c7948d6-0bec-4391-8b88-acf67e1d28c4.png)

Setup on https://fin.kujira.network/ using KUJI/USK and KUJI/DEMO using Pure Market-making:

• Able to start PMM with KUJI/DEMO, observed:

  • Orders are successfully created and canceled
  • Cancelation would have delays of 10s and sometimes orders get stuck and not recognized by the client when start again (test in progress)

When we start or stop the strategy we are dispatching a process that will try to cancel all orders from the user, which may take some seconds indeed. Before we are canceling all and claiming all funds (but we are now only canceling all order only, since we needed to remove all specific routes to conform to the requested changes).
Even if this process fail (although may take some seconds) it should be ok since we have a try catch in the code for that.
We have there on our cancell_all_orders method on our hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py file.

• However for the logs panel does not seem to be displaying correctly
  

• Setup tight spreads for order filled events
  

  • ⚠️ The number of trades from client is half the value of Filled Orders from https://fin.kujira.network/

We don't know too much about this part and we are investigating if an issue is happening there.
(
Some possibilities are:
it maybe have been from a previous test or filled order;
there's a problem somehow in our code, although we always place orders using the inherited methods, so no order is placed from outside, so it's a bit strange;
the Kujira FIN is lazy and it's not reporting correctly;
HB has some issue with the trading history
)

* Also. we would like to know if we need to `Claim all`?

Yes, it's actually better to "Enable the Auto Widthdraw" because we don't have a specific route to automatically claim / withdraw orders anymore. This process is basically to return the funds of filled orders to the user wallet so they can be used again for market making.

The best way to accompanying the orders I'd say it would be using the "History" tab when the automatic claim / withdraw is enabled.

When starting the client, getting unhandled error in background for KUJI/USDK

This error is mainly because this market does not exists, there's a typo ;)

The correct market would be KUJI/USK. We usually use KUJI/DEMO but from our integration tests KUJI/USK, KUJI/DEMO and DEMO/USK can be used. But from the Kujira FIN frontend perspective, it seems KUJI/DEMO is the one that works better.

2023-07-14 19:45:22,869 - 16657 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - cancel_all_orders: start
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - _update_markets: start
2023-07-14 19:45:22,871 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets request:
 "{
  "connector": "kujira",
  "chain": "kujira",
  "network": "testnet",
  "trading_pair": "KUJI-USDK"
}".
2023-07-14 19:45:22,919 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets response:
 "{
  "network": "testnet",
  "timestamp": 1689335122875,
  "latency": 0.001,
  "markets": {}
}".
2023-07-14 19:45:22,920 - 16657 - hummingbot.client.hummingbot_application - ERROR - Error canceling outstanding orders.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/hummingbot_application.py", line 256, in _cancel_outstanding_orders
    cancellation_results = await market.cancel_all(kill_timeout)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 723, in cancel_all
    return await self._api_data_source.cancel_all(timeout_seconds)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 789, in cancel_all
    return await self.cancel_all_orders()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 388, in cancel_all_orders
    self._check_markets_initialized() or await self._update_markets()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 723, in _update_markets
    self._market = self._markets[self._trading_pair]
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/dotmap/__init__.py", line 99, in __getitem__
    return self._map[k]
KeyError: 'KUJI-USDK'

Run stop command getting Error canceling outstanding orders and client is stuck, we have to run exit -f so we can start anew.

This problem happened because the market is non-existent (like I explained above) and then this dispatched a fatal error when initialized the strategy.

2023-07-14 19:45:22,869 - 16657 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - cancel_all_orders: start
2023-07-14 19:45:22,870 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - _update_markets: start
2023-07-14 19:45:22,871 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets request:
 "{
  "connector": "kujira",
  "chain": "kujira",
  "network": "testnet",
  "trading_pair": "KUJI-USDK"
}".
2023-07-14 19:45:22,919 - 16657 - hummingbot.connector.gateway.clob_spot.data_sources.kujira.kujira_api_data_source.KujiraAPIDataSource - DEBUG - get_clob_markets response:
 "{
  "network": "testnet",
  "timestamp": 1689335122875,
  "latency": 0.001,
  "markets": {}
}".
2023-07-14 19:45:22,920 - 16657 - hummingbot.client.hummingbot_application - ERROR - Error canceling outstanding orders.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/client/hummingbot_application.py", line 256, in _cancel_outstanding_orders
    cancellation_results = await market.cancel_all(kill_timeout)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 723, in cancel_all
    return await self._api_data_source.cancel_all(timeout_seconds)
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 789, in cancel_all
    return await self.cancel_all_orders()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 388, in cancel_all_orders
    self._check_markets_initialized() or await self._update_markets()
  File "/Users/rapcomia/github/hummingbot/6399/hummingbot/connector/gateway/clob_spot/data_sources/kujira/kujira_api_data_source.py", line 723, in _update_markets
    self._market = self._markets[self._trading_pair]
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/dotmap/__init__.py", line 99, in __getitem__
    return self._map[k]
KeyError: 'KUJI-USDK'

So for testnet, are we ok with using KUJI/USK for running tests? thank you 🙇🏼

Yes, we think so, although I'd suggest KUJI/DEMO.

Steps to reproduce:

• Connect to kepler wallet and use testnet
• Setup simple PMM for both KUJI\USK and KUJI\DEMO
• Observe connector behavior

logs_pmm-kujira.log

By the way, we have noticed a strange behavior,
because we don't have other DEXes settedu-p here we cannot confirm,
but it seems the start and stop of the strategies does not work well when the strategy stops "too" fast.

What I mean is, initially we had a method to automatically cancel and claim all orders in the end, then, because of the changes removing our specific routes, we removed this. We then noticed this behavior of the start and stop commands having problems. When we added the cancel all orders again (and then it took longer to stop) the start and stop commands started to work better again ^^

It's a bit strange, but because we haven't changed anything from the PMM strategy and we don't have errors in our log (when it was happening fast) we couldn't really know what was happening. Maybe it's a reflex of some new incorporated changes to the dev branch.

If we have any news about the trading count I'm going to comment here..

[danilo-silva-funttastic]

Please take a look when you have time. We might be good to go with these current changes.

[danilo-silva-funttastic]

@cardosofede, can you take a look and possible approve this PR of ours here for the Gateway as well? ;)

[danilo-silva-funttastic]

@cardosofede, as requested we've squashed our commits to only one.

[danilo-silva-funttastic]

Hey @fengtality, can you please review our PR when you have time?
We've done all of the requested changes, our tests are passing, and from our manual tests everything should be good.
We've also squashed the commits to only one as requested.

[rapcmia]

LGTM

• Setup this PR with the Feat / Kujira hummingbot#6399
• Connection successful
• Ran endpoint tests using thunderclient
  • system/connectors -> Kujira available
  • wallet/{get, add, remove}
  • chain/{status, config, balances, poll, tokens}
• Results noted on the client PR see link above
• Succesfully built docker image and setup using hummingbot-gateway-compose

[danilo-silva-funttastic]

A disclaimer about why our build check is not passing (@fengtality, @rapcmia, @cardosofede).

We've done some tests here and when modifying the .github/workflows/workflow.yml to allow the build to run all unit tests, the current development branch of HB seems not be working as well (information in the end).

Our best guess is the following: Our connector (alongside to its specific implementations) also changed some core files (to add support to the new connector). This then made the diff of the github workflow there to find a lot of test files from other connectors as well and then tried to run all of them. And it seems they are failing (below are the results when running from the current development branch of HB without any Kujira). Maybe this is due to the memory needed for the process to run there, but I'm not sure. In any case, it seems that are not related to us since when we run our build (with our changes) the failing tests are not ours. And our unit tests are passing.

Here's an output of the failing tests from the modified workflow to allow all unit tests to run without trying to find the related tests. The failing tests below were run without any kujira code, but only the raw HB development branch:

FAIL test/clob/clob.routes.test.ts (7.03 s)
FAIL test/chains/ethereum/ethereum.routes.test.ts (7.85 s)
FAIL test/connectors/openocean/bsc.openocean.routes.test.ts
FAIL test/connectors/openocean/avalanche.openocean.routes.test.ts
FAIL test/connectors/openocean/arbitrum.openocean.routes.test.ts
FAIL test/connectors/openocean/optimism.openocean.routes.test.ts
FAIL test/connectors/openocean/harmony.openocean.routes.test.ts
FAIL test/connectors/openocean/ethereum.openocean.routes.test.ts
FAIL test/chains/algorand/algorand.routes.test.ts (6.035 s)
FAIL test/connectors/openocean/polygon.openocean.routes.test.ts (6.61 s)
FAIL test/connectors/openocean/cronos.openocean.routes.test.ts (14.763 s)
FAIL test/connectors/uniswap/uniswap.routes.test.ts (7.483 s)
FAIL test/connectors/sushiswap/ethereum.sushiswap.routes.test.ts (6.412 s)
FAIL test/connectors/defira/defira.routes.test.ts (8.485 s)
FAIL test/connectors/pancakeswap/pancakeswap.routes.test.ts (7.782 s)
FAIL test/connectors/mad_meerkat/mad-meerkat.routes.test.ts (7.581 s)
FAIL test/connectors/pangolin/pangolin.routes.test.ts (13.112 s)
FAIL test/connectors/quickswap/quickswap.route.test.ts (10.117 s)
FAIL test/connectors/vvs/vvs.routes.test.ts (8.614 s)
FAIL test/chains/xdc/xdc.routes.test.ts (9.257 s)
FAIL test/connectors/xsswap/xsswap.routes.test.ts (15.451 s)
FAIL test/connectors/traderjoe/traderjoe.test.ts (5.796 s)
FAIL test/chains/harmony/harmony.routes.test.ts (12.439 s)
FAIL test/chains/avalanche/avalanche.routes.test.ts (17.982 s)
FAIL test/chains/binance-smart-chain/binance-smart-chain.routes.test.ts (9.546 s)

The diff of the modified .github/workflows/workflow.yml is the following:

diff --git a/.github/workflows/workflow.yml b/.github/workflows/workflow.yml
index 9f1c6a4..8f31800 100644
--- a/.github/workflows/workflow.yml
+++ b/.github/workflows/workflow.yml
@@ -6,9 +6,9 @@ name: ci
 # or API.
 on:
   push:
-    branches: [main, development, 'refactor/unit_tests**', 'epic/**']
+    branches: [main, development, 'refactor/unit_tests**', 'epic/**', 'fix/workflow']
   pull_request:
-    branches: [main, development, 'refactor/unit_tests**', 'epic/**']
+    branches: [main, development, 'refactor/unit_tests**', 'epic/**', 'fix/workflow']
     types: [ready_for_review, opened, synchronize, reopened]

 jobs:
@@ -28,7 +28,6 @@ jobs:
             !*.sh
       - name: Check if gateway files are modified
         id: check_files
-        if: env.GIT_DIFF
         run: |
           echo ${{ env.GIT_DIFF }}
           echo "::set-output name=is_set::true"
@@ -36,7 +35,6 @@ jobs:
   build_gateway:
     name: Gateway build + unit tests
     needs: run_gateway
-    if: github.event.pull_request.draft == false && needs.run_gateway.outputs.is_set == 'true'
     runs-on: ubuntu-latest

     steps:
@@ -94,14 +92,11 @@ jobs:
           sed -i 's/https:\/\/evm.cronos.org/http:\/\/127.0.0.1:8545\//g' ./conf/cosmos.yml

       - name: Run unit test coverage
-        if: github.event_name == 'pull_request'
         shell: bash
         run: |
           git fetch --all -q
-          git checkout -b $GITHUB_SHA
-          DIFF_FILES=`git diff --name-only origin/$GITHUB_BASE_REF ./src/`
-          ./node_modules/.bin/jest --listTests  --findRelatedTests ${DIFF_FILES//$'\n'/ }
-          NODE_OPTIONS=--max-old-space-size=10240 node ./node_modules/.bin/jest --runInBand --coverage --findRelatedTests ${DIFF_FILES//$'\n'/ }
+          ./node_modules/.bin/jest --listTests
+          NODE_OPTIONS=--max-old-space-size=10240 node ./node_modules/.bin/jest --runInBand --coverage
 #          git diff origin/$GITHUB_BASE_REF | yarn diff-test-coverage -c ./coverage/lcov.info -t lcov -b 50 -l 60

   docker_build_and_push:
@@ -116,7 +111,7 @@ jobs:
         uses: docker/setup-buildx-action@v1

       - name: Login to DockerHub
-        uses: docker/login-action@v1
+        uses: docker/login-action@v1
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}

[fengtality]

Hey @danilo-silva-funttastic this connector looks very non-standard compared to the other ones that community devs are submitting into Gateway. It doesn't make sense to introduce new design patterns for fetching tokens and I don't know why a cache is needed. I need you to conform this connector to the standards that other chain/DEX connectors in Gateway are using.

[fengtality]

Please remove unused libraries like crypto-js

[danilo-silva-funttastic]

Unused dependency removed.

[fengtality]

Importing contract addresses via a library like this import contracts from 'kujira.js/src/resources/contracts.json'; is not secure, since it doesn't allow users to see what addresses they are interacting with and they can be modified without them seeing it. Remove this dependency and put the contract addresses as parameters in kujira.yml or in the Kujira connector code.

[danilo-silva-funttastic]

This dependency actually cannot be removed. Kujira doesn't have an npm package and we need to import it's library from it's github.

[danilo-silva-funttastic]

Regarding the kujira.js/src/resources/contracts.json we are actually taking care of that. By default we indeed import the contracts from the library, but the user also has options to use a local file or an external url. Also, for the tokens and the markets we have an important mechanism to allow the users to choose which tokens or markets they want to use, this avoids making the connector takes to long to initiate and loading too many markets or tokens; and also enables the user to use only the ones they specified. We can change the behavior to use, by default the contract from the url (we've done this way because for a default behavior its more reliable since an external url might change but library file will always be available).

Here's where the user can configure it:

tokens:
  url: # empty means using the default
#  url: /root/gateway/conf/connectors/kujira/tokens.json
#  url: https://raw.githubusercontent.com/Team-Kujira/kujira.js/master/src/resources/tokens.json
  allowed:
  disallowed:
  resolutionStrategy: markets
markets:
  url: # empty means using the default
#  url: /root/gateway/conf/connectors/kujira/contracts.json
#  url: https://raw.githubusercontent.com/Team-Kujira/kujira.js/master/src/resources/contracts.json
  allowed:
  disallowed:

[fengtality]

Gateway is supposed to be stateless middleware that the Hummingbot client uses to send transactions - this is why Hummingbot has order tracking support. Why are you using a cache?

[danilo-silva-funttastic]

Gateway still remains stateless, cache was introduced to avoid loading some heavy data all over again every time they are requested. For example, when a user fetch information from a market, since it comes from a json file or an external url and some transformation happens behind the scenes, we cache the markets for immediate use and quick responses.

This is also configurable by the user, if he/she wants to disable it they can use this option in the kujira.yml

cache:
  marketsData: 3600 # in seconds
  markets: 3600 # in seconds

We tried to make our kujira.yml very simple and intuitive to use. All connectors have their own world and then request different yml approaches.

[fengtality]

Kujira tokens should be fetched just like all other chains, i.e. each network in conf/kujira.yml defines tokenListSource and tokenListUrl (FILE) and the token list is saved in conf/lists.

[danilo-silva-funttastic]

This is something that has changed while we were developing so it is working differently indeed.
But although HB now has a standard for the tokenList there's no standard for the marketList, which is very needed too.
We can change the names to tokenListSource, tokenListUrl and save it there on conf/lists, but we still need a place to save the markets list as well.

In any case, those things related to the kujira.yml if they are not wrong or causing problems, I'd like to propose to leave them this way, as you know each connector might have very different configuration files and for the tokenListType, from the value informed, we already know if we should fetch it from an url or from a file, so it's not really needed. As for now HB doesn't have a section for the markets information to be saved.

Would that work for you @fengtality? I think the Kujira users, as long as our yml has all the explanations, will not have problems setting it up.

[danilo-silva-funttastic]

@fengtality

We have replaced the mechanism responsible for the token loading and we are now using the tokenListSource and tokenListUrl as requested.

We have also removed the caching mechanism and the libraries from the package.json.

Also simplified and removed some other functionalities.

We might be good to know, please review when you have time.

[danilo-silva-funttastic]

Keeping the branch up to date to the HB development branch.

[danilo-silva-funttastic]

Keeping the code up-to-date to the HB development branch.

[fengtality]

Looks good overall! Please respond to the 2 comments here.

Also, please move the Kujira tests from /test/ to /test-bronze folder, since they should not be included in Github workflow.

[danilo-silva-funttastic]

Conforming the requested changes.

[danilo-silva-funttastic]

@fengtality, all of the requested changes were made.

We are now using an NPM library, the unit tests were moved to the test-bronze folder and the validation inside the src/clob/clob.validators.ts has been reverted.

[danilo-silva-funttastic]

Hey @fengtality, thanks for approving our PR! :)

I was double checking here why the build failed and the offending test is not ours:

test/connectors/pancakeswap/pancakeswap.test.ts

I have also ran it here on my machine and it has passed.
So it's that kind of that situation of some tests sometimes do not pass out of random.

We have also done a previous exhaustive study (details here on this comment) and showed that in the current HB development branch if we run all tests they don't usually pass.
From the current github workflow, there's a mechanism to try to decide which test to run based on the files changed. But because we've changed some few core files to enable the connector to be integrated, this then dispatch a bunch of other tests from other connectors, which some times fail in return. This then creates a difficult scenario to have a green build.

Please kindly let us know how we can proceed.