Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Partinfo API #346

Closed
kasbah opened this issue Feb 22, 2019 · 70 comments
Closed

Use Partinfo API #346

kasbah opened this issue Feb 22, 2019 · 70 comments
Assignees
Labels
discussion Discution about implementation and new features. feature-request New features resquested.
Milestone

Comments

@kasbah
Copy link
Contributor

kasbah commented Feb 22, 2019

As I mentioned on the forum I'd be happy for KiCost users to make use of the Partinfo API.

I am happy for people working on BOM tools, especially KiCost users and developers, to use (but not abuse) the Partinfo endpoint right now, specifically: https://dev-partinfo.kitspace.org/graphql.

It's currently an open endpoint so anyone can access it. Docs are a bit thin but you are able to browse the schema using the "Documentation Explorer" on the right on the above URL and you can jump on our chat or the GitHub issue tracker if you have any questions. You don't need any special GraphQL client to use it, here is the example query using curl:

curl  -H 'Content-Type: application/json' -X POST -d '{"query": "{ part(mpn: {part: \"NE555P\", manufacturer: \"Texas Instruments\"}) { datasheet description type offers { sku { vendor part } prices { USD EUR GBP SGD } } } }"}' https://dev-partinfo.kitspace.org/graphql | python -m json.tool

To cover the API costs when Octopart does ask for payment I suggest we pool money in opencollective.com/kitspace. I estimate that $100 a month will be enough to cover current KiCost users (but we will see of course).


Some more explanation of the current Partinfo schema.

Currently there are 3 query types and all of the return Part types. As with the Octopart API the Part types contains an array of Offer types from different retailers with prices. E.g.

part(mpn: Mpn, sku: Sku): Part

Returns a single Part that matches a manufacturer part number (Mpn) or retailer stock keeping unit (Sku). This is considered a match. You can also batch multiple part matches requests and get back multiple parts.

match(parts: [MpnOrSku]): [Part]

We can also search and get back multiple parts.

search(term: String): [Part]

Here free form text is parsed by Electro Grammar and turned into parametric Octopart searches and also matched against the Common Parts Library.

@kasbah kasbah changed the title Switch KiCost to using Partinfo API Use Partinfo API Feb 22, 2019
@hildogjr hildogjr added feature-request New features resquested. discussion Discution about implementation and new features. labels Mar 16, 2019
@hildogjr
Copy link
Owner

@kasbah, the workaround using your proxy is finished (and other improvements on the spreadsheet). I am waiting @devbisme / @xesscorp for checking.

I didn't undestand how to use your API, could you explain with some Python / address that I could direct pass to my browner ?

@kasbah
Copy link
Contributor Author

kasbah commented Mar 16, 2019

So the API uses GraphQL. There are some client libraries available for Python that may make things easier but the basic principle is that you make a query defining the data you want back for all requests and use the variables field to input different parts.

Here is an example using requests that gets back all the available info about NE555P. Below you could define the input variable to another part you want information about and also reduce the query to just the information you want.

import requests

query = """
    query ($input: MpnInput!) {
      part(mpn: $input) {
        mpn {
          manufacturer
          part
        }
        type
        datasheet
        description
        image {
          url
          credit_string
          credit_url
        }
        specs {
          key
          name
          value
        }
        offers {
          sku {
            vendor
            part
          }
          description
          moq
          in_stock_quantity
          stock_location
          image {
            url
            credit_string
            credit_url
          }
          specs {
            key
            name
            value
          }
          prices {
            GBP
            EUR
            USD
          }
        }
      }
    }
"""

variables = '{"input": {"part": "NE555P", "manufacturer": "Texas Instruments"}}'

r = requests.post("https://dev-partinfo.kitspace.org/graphql", {"query": query, "variables": variables})

print(r.json())

Here is the same example in the interactive GraphiQL interface where it may be easier to play around a bit.

@hildogjr
Copy link
Owner

hildogjr commented Mar 18, 2019

Interesting, start to understanding. Five comments:

  1. How can I request more than one part? Or have I to request one-by-one?
  2. How to just specify the mpn code and not the Manufacture name? (I tried variables = '{"input": {"part": "NE555P"}}'
  3. The datasheet result was 'datasheet': 'https://????.com/click/track?ak=????&ct=datasheets&sig=????&at=physicalpart&sid=????&ppid=?????&hlid=??????????'. Please, resolve this link (and others) before pass through the API. And is missing some URL for the product page (at each distributor/vendor).
  4. For KiCost will be interest specify the vendors that I want the answers.
  5. What means moq at offers?

For now, this answers and the bellow configuration

query ($input: MpnInput!) { 
      part(mpn: $input) {   
                mpn {     part    } 
                datasheet
                specs {   key   value   }
                offers { 
                        sku {        vendor        part      } 
                        moq
                        in_stock_quantity
                        prices {        USD      } 
                 }
       }
  }

will restore the KiCost main capabilities, the other infos stay for future implementation.

@kasbah
Copy link
Contributor Author

kasbah commented Mar 19, 2019

Thanks for looking into it.

  1. There is also a match query where you enter multiple parts (and a search query for text input)
  2. You currently have to use an empty string {"part": "NE555P", "manufacturer": ""}, we should fix that
  3. The datasheet result

We can work on that I guess, is it important?

  1. specify the vendors

Makes sense, that requires a bit of work on the Partinfo site. We are currently restricted to the 5 vendors supported by Kitspace.

  1. moq stands for minimum-order-quantity, the smallest quantity you can buy of a particular part

FYI Partinfo also doesn't currently have pagination thus max search results returned are currently 20, is that a problem?

@hildogjr
Copy link
Owner

  1. I still need to learn how to use, I am planning it to future Automatic specify generic components #17;
  2. Yes, maybe passing just {"part": XXX} , Kitspace could understand a empty manufacturer string;
  3. May be just my opinion. But I would like to remove each trace of dependence in the KiCost;
  4. Witch vendors are now supports? (KiCost was using Digikey, Mouser, Farnell, Newark, RS, TME and Arrow. It is interesting provide these at next KiCost update);
  5. Interesting information! I am planning to use.

It is not a problem just return 20 by time. KiCost just have to ask twice...

I am a quite of time now, I already update something at spreadsheet.py but the big works still need to be done at distributors folder (using dist_octopart.pyas template).

@hildogjr
Copy link
Owner

@xesscorp / @devbisme I just finished some improvements at spreadsheet.py on the branch Octopart.
Now this branch is froze and we can move into Partinfo API on the main branch.

@hildogjr
Copy link
Owner

hildogjr commented Mar 25, 2019

@kasbah, I am having problem with the API Mouser response (testing in the query system yet) for the part
IPW60R160P6FKSA1
CGA9N2X7R2A105M230KA

So far I checked, it appear be related to the price break, even on the web page those appear different. But I think is not your code layer.

@kasbah
Copy link
Contributor Author

kasbah commented Mar 26, 2019

Sorry, what is the problem exactly?

@hildogjr
Copy link
Owner

For example IPW60R160P6FKSA1 returned as "0 in stock".
But the web page says the opposite
https://www.mouser.it/ProductDetail/Infineon-Technologies/IPW60R160P6FKSA1?qs=sGAEpiMZZMshyDBzk1%2FWizrSbRfT%2Fm2Uc4Oi9zmiFXNiGpJHGELpdQ==

@kasbah
Copy link
Contributor Author

kasbah commented Mar 26, 2019

Just checked, that's down to Octopart unfortunately, they have the wrong stock information for that part at the moment.

On a separate issue:

It is not a problem just return 20 by time. KiCost just have to ask twice...

There is no way to get more than 20 results even if you ask twice.

@hildogjr
Copy link
Owner

I imagined that. So don't worry, nothing we can do for this.

There is no way to get more than 20 results even if you ask twice.
If I have a list of 35 items, I can't request 20, wait and request again with the other 15? KiCost do this with Octopart.

@kasbah
Copy link
Contributor Author

kasbah commented Mar 26, 2019

In the future we can figure out an additional data source for Mouser. Partinfo does support the Farnell API as well, and could improves the stock information for that, but the API limit prevents me from turning it on at the moment. Hopefully I will be able to fix that soon.

No, currently it's a hard limit of 20 items for the search and match queries. part queries only return a single result anyway.

@hildogjr
Copy link
Owner

@kasbah, for use the match query I:

  1. Changed the query to:

    query ($input: MpnOrSku!) {
    match(parts: $input) {

  2. The variable to:

    {
    "input": {
    "mpn": "RNCP1206FTD33R2",
    "sku": ""
    }
    }

What I am doing wrong?

And 2 question:

  1. How to request more than one match?
  2. The search query? Is it possible to request more than one search by request? Could you provide me some example of this two?

@kasbah
Copy link
Contributor Author

kasbah commented Mar 30, 2019

You need a list with the proper structure as input. Here you go. I don't think there is a limit on the size of the match list at the moment.

search is currently one per request. Here is an example search query.

@hildogjr
Copy link
Owner

hildogjr commented May 31, 2019

@kasbah , please gimme a tip / have a look at line 193 of api_partinfo_kitspace.py.

I have the query type

query ($input: [MpnOrSku]!){ match(parts: $input) {mpn{manufacturer,part},type,datasheet,description,image{url,credit_string,credit_url},specs{key,name,value},offers{sku{vendor,part},description,moq,in_stock_quantity,stock_location,image{url,credit_string,credit_url},specs{key,name,value},prices{GBP,EUR,USD}}} }

the components query

{ "input": [{"mpn":{"part":"RMCF0805JT10K0","manufacturer":""}},{"mpn":{"part":"1375820-3","manufacturer":""}},{"mpn":{"part":"1N4728ATR","manufacturer":""}},{"mpn":{"part":"ECE-A1VKA100B","manufacturer":""}},{"mpn":{"part":"C0805C105K5RACTU","manufacturer":""}},{"mpn":{"part":"RMCF0805JT150R","manufacturer":""}},{"mpn":{"part":"MOC3021S-TA1","manufacturer":""}},{"mpn":{"part":"RMCF0805JT4K70","manufacturer":""}},{"mpn":{"part":"640456-4","manufacturer":""}},{"mpn":{"part":"ECQ-E4225KF","manufacturer":""}},{"mpn":{"part":"C0805C224K5RACTU","manufacturer":""}},{"mpn":{"part":"RMCF0805JT330R","manufacturer":""}},{"mpn":{"part":"MC7805CDTG","manufacturer":""}},{"mpn":{"part":"ISO1042BDWR","manufacturer":""}},{"mpn":{"part":"ISO1042BDWR","manufacturer":""}},{"mpn":{"part":"BC847BLT1G","manufacturer":""}},{"mpn":{"part":"LL4148","manufacturer":""}},{"mpn":{"part":"RMCF0805JT120R","manufacturer":""}},{"mpn":{"part":"BTA204-600E","manufacturer":""}},{"mpn":{"part":"RMCF0805JT470R","manufacturer":""}}]}

The code is summarized to

import requests

query="query ($input: [MpnOrSku]!){ match(parts: $input) {mpn{manufacturer,part},type,datasheet,description,image{url,credit_string,credit_url},specs{key,name,value},offers{sku{vendor,part},description,moq,in_stock_quantity,stock_location,image{url,credit_string,credit_url},specs{key,name,value},prices{GBP,EUR,USD}}} }"

variables = '{input:[{"mpn":{"part":"RMCF0805JT10K0","manufacturer":""}},{"mpn":{"part":"1375820-3","manufacturer":""}},{"mpn":{"part":"1N4728ATR","manufacturer":""}},{"mpn":{"part":"ECE-A1VKA100B","manufacturer":""}},{"mpn":{"part":"C0805C105K5RACTU","manufacturer":""}},{"mpn":{"part":"RMCF0805JT150R","manufacturer":""}},{"mpn":{"part":"MOC3021S-TA1","manufacturer":""}},{"mpn":{"part":"RMCF0805JT4K70","manufacturer":""}},{"mpn":{"part":"640456-4","manufacturer":""}},{"mpn":{"part":"ECQ-E4225KF","manufacturer":""}},{"mpn":{"part":"C0805C224K5RACTU","manufacturer":""}},{"mpn":{"part":"RMCF0805JT330R","manufacturer":""}},{"mpn":{"part":"MC7805CDTG","manufacturer":""}},{"mpn":{"part":"ISO1042BDWR","manufacturer":""}},{"mpn":{"part":"ISO1042BDWR","manufacturer":""}},{"mpn":{"part":"BC847BLT1G","manufacturer":""}},{"mpn":{"part":"LL4148","manufacturer":""}},{"mpn":{"part":"RMCF0805JT120R","manufacturer":""}},{"mpn":{"part":"BTA204-600E","manufacturer":""}},{"mpn":{"part":"RMCF0805JT470R","manufacturer":""}}]}'

r = requests.post("https://dev-partinfo.kitspace.org/graphql", {"query": query, "variables": variables})

print(r.json())

That works at your GraphiQL platform. But I am getting response 400 (Bad request) from my Python code.

@hildogjr
Copy link
Owner

Forgot @kasbah, it was just missing " in "input".

@hildogjr
Copy link
Owner

hildogjr commented Jun 1, 2019

@kasbah, Are the server offline? I am getting "504 Gateway Timeout" (also with https://dev-partinfo.kitspace.org/graphql)

@kasbah
Copy link
Contributor Author

kasbah commented Jun 1, 2019

Yes, there was an issue. Should be fixed now. Thanks for letting me know.

hildogjr added a commit that referenced this issue Jun 2, 2019
@hildogjr
Copy link
Owner

hildogjr commented Jun 2, 2019

Some alpha code is ready but still need some fix to the cases the PartInfo return no available data for one specific part in one distributor.

@kasbah
Copy link
Contributor Author

kasbah commented Jun 2, 2019

Can you give an example?

@hildogjr
Copy link
Owner

hildogjr commented Jul 5, 2019

@xesscorp, I think that I have fixed the "add to BOM list" Windows issue. Missing the context menu.

@xesscorp
Copy link
Collaborator

xesscorp commented Jul 7, 2019

I downloaded the latest KiCost and ran it under Windows 7 with KiCad 5.1.2.

When I ran kicost --setup, I got this output:

KiCost identified at 'c:\xesscorp\kicad\tools\kicost\kicost', proceding with it configuration in file 'C:\Users\DEVB\AppData\Roaming\kicad'...
GUI requirements (wxPython) identified.
Creating app shortcuts...
Check your desktop for the KiCost shortcut.
Creating OS context integration...
GUI requirements (wxPython) identified.
Failed to create KiCost OS context menu integration.  <== ???
Setting the GUI to display the NEWS message...
The will display the NEWS message on first staturp.
Creating KiCad integration...
Failed to add KiCost as KiCad BOM plugin.  <== ???
KiCost setup configuration finished.

After that, I ran KiCad and EESCHEMA. KiCost was in the BOM list and it started running when I clicked on it. So I'm not sure if it actually failed to be added. I also saw KiCost in the context menu when I right-clicked on an .xml file.

It's possible KiCost had already been added before I ran the --setup, so I decided to run kicost --unsetup to remove them and got the following:

Removing BOM plugin entry from Eeschma configuration...
Traceback (most recent call last):
  File "C:\Python37-64\Scripts\kicost-script.py", line 11, in <module>
    load_entry_point('kicost', 'console_scripts', 'kicost')()
  File "c:\xesscorp\kicad\tools\kicost\kicost\__main__.py", line 187, in main
    kicost_unsetup()
  File "c:\xesscorp\kicad\tools\kicost\kicost\kicost_config.py", line 485, in kicost_unsetup
    remove_bom_plugin_entry(kicad_config_path, 'KiCost')
  File "c:\xesscorp\kicad\tools\kicost\kicost\kicost_config.py", line 167, in remove_bom_plugin_entry
    if re.findall(name, plugin[1], re_flags):
  File "c:\python37-64\lib\re.py", line 223, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or bytes-like object

If you can get the --unsetup to work, I will try again.

hildogjr added a commit that referenced this issue Jul 8, 2019
@hildogjr
Copy link
Owner

hildogjr commented Jul 8, 2019

@xesscorp, I had fix the Win issue related do path named at the add_bom_plugin_entry function and forgot at remove_bom_plugin_entry. Now I propagate the changes. Please, check again.

When you use the context-menu in a .xml file, did the GUI open with the specified file?
Can you reproduce the "Adding KiCost to the Context Menu (Windows Only)" section at https://xesscorp.github.io/KiCost?

After this, I will release on PyPI.

@hildogjr
Copy link
Owner

@xesscorp, I create the documentation by your make file (I don't know how to publish it after). Please check it and the --setup/--unsetup commands, I will release on PyPI after.

@xesscorp
Copy link
Collaborator

--unsetup still fails:

# kicost --unsetup
Removing BOM plugin entry from Eeschma configuration...
Traceback (most recent call last):
  File "C:\Python37-64\Scripts\kicost-script.py", line 11, in <module>
    load_entry_point('kicost', 'console_scripts', 'kicost')()
  File "c:\xesscorp\kicad\tools\kicost\kicost\__main__.py", line 187, in main
    kicost_unsetup()
  File "c:\xesscorp\kicad\tools\kicost\kicost\kicost_config.py", line 504, in kicost_unsetup
    remove_bom_plugin_entry(kicad_config_path, 'KiCost')
  File "c:\xesscorp\kicad\tools\kicost\kicost\kicost_config.py", line 171, in remove_bom_plugin_entry
    search = plugin[1].replace("\\",'/')
AttributeError: 'Symbol' object has no attribute 'replace'

@hildogjr
Copy link
Owner

hildogjr commented Jul 15, 2019

Fixed the issue and already in
https://pypi.org/project/kicost/

I think that I also fix #289

@xesscorp, release or gimme so guidance to release/update https://xesscorp.github.io/KiCost.

@hildogjr
Copy link
Owner

@kasbah, KiCost with PartInfo API is released https://forum.kicad.info/t/new-kicost-release/8074/2
Please, keep us updated about the server use and some statistics (for example country of access, ...). If necessary, we can add a "&tool=kicost" in the request link to help you with the statistics.

@kasbah
Copy link
Contributor Author

kasbah commented Jul 31, 2019

FYI there have been over 700 octopart requests resulting from KiCost since I put the service up again.

@xesscorp
Copy link
Collaborator

Over what time period?

@hildogjr
Copy link
Owner

hildogjr commented Jul 31, 2019

Do you have some statistic of how much of them was new (not replicated or already cached by you) or the traffic day-by-day?
Because, the code take a full BOM and broke into maximum 25 parts request. Because I was testing (I ran the full repository with 35 different BOMs), I may be responsible by some replicated access.

@kasbah
Copy link
Contributor Author

kasbah commented Jul 31, 2019

Since 27th of June. This only counts cache misses i.e. when it actually makes requests to Octopart. Oh, and it actually also includes people using https://bom-builder.kitspace.org.

It does seem like a lot. I should set up some more sensible logging.

@hildogjr
Copy link
Owner

hildogjr commented Jul 31, 2019

I think, it will be interesting for your:

  1. Add "&tool=kicost" at KiCost request (or something like that). And block if this string is empty, to force the users to use the last version;
  2. I could also add some "&request_counter=#" to inform you the case of sequential access (that case that the BOM is broken in multiple requests). Making #=1, 2, 3... by the sequential attempt.
  3. Get some statistics using the region/country of access (is this possible?).
  4. Shutdown your proxy, if it is online yet (I don't know if some user is using that).

@kasbah
Copy link
Contributor Author

kasbah commented Jul 31, 2019

  1. I think what I'll do in the future is make people generate a key by some sort of sign up.
  2. I think caching should be coping with this already
  3. It's possible but I don't have time right now
  4. Proxy is down since June 22nd

Let's just keep an eye on it for now, I don't have a lot of time to spend on this unfortunately.

@hildogjr
Copy link
Owner

I imagine that "1" was inevitable to the KiCost future.
Because this, the GUI already support save parameters (and I was using this to to save the Octopart key).

@hildogjr
Copy link
Owner

hildogjr commented Aug 6, 2019

@kasbah, how to search by the distributor catalogue number? (Is it possible in the actual API?)
KiCost prioritize the distributor catalogue number to disambiguation. For example:
part-link: https://www.digikey.com/products/en/sensors-transducers/humidity-moisture-sensors/529?k=dht11
manf#: 386
digikey#: 1528-1228-ND

If I search only by 386, I get the not correct part. So, in my schematic I fill the two information and KiCost prioritize digikey# number under manf# in the Digikey distributor. Using just manf# for other distributors that not Digikey.

Is it possible something similar? (Allow my to get a specific part for the catalog number in one distributor?)

@kasbah
Copy link
Contributor Author

kasbah commented Aug 6, 2019

Use sku instead of mpn. By the way, you also get the correct response if you add "Adafruit" as manufacturer.

@hildogjr
Copy link
Owner

hildogjr commented Aug 6, 2019

I am reopening until add this feature.

@hildogjr hildogjr reopened this Aug 6, 2019
hildogjr added a commit that referenced this issue Aug 11, 2019
hildogjr added a commit that referenced this issue Aug 11, 2019
@hildogjr
Copy link
Owner

hildogjr commented Aug 12, 2019

I was already using sku, I just had to change the query logic to prioritize "stock codes".

@kasbah, could you give me some direction about use https://github.com/kitspace/electro-grammar on Python in local machine (without unnecessary query from your server)?
Can I get Antlr4 from pip? There are other requirements to be cross platform (Win, Linux, ...)?
It is something moving foward to #17 and #4.

@kasbah
Copy link
Contributor Author

kasbah commented Aug 12, 2019

The Antlr4 port is pretty much abandoned. I did have an earlier experiment to port to Python using Lark but it's not finished either. The best way to go about finishing it would probably be to port the test-suite to Python first of all.

@hildogjr
Copy link
Owner

@kasbah, I was improving some behaviors and checking about the query responses. I am sharing a log of the full batch test (I always used the match query):

See the 'TPS71725QDRV' manufacture part number, for example, it's return as "not found" because, at least on Digikey the complete code is 'TPS71725QDRVRQ1'. Previous versions of KiCost dealt with this by some string match but I don't now if is possible now.

As far I checked is the same for all the reported "not found" parts that not the '{'sku': {'part': '694-9374', 'vendor': 'RS'}}', for some reason your API is return empty price.

test.log

@kasbah
Copy link
Contributor Author

kasbah commented Sep 16, 2019

Hey sorry I missed your message. Can you report future issues to the Partinfo GitHub please rather than this hugely long thread. I believe the first issue is already reported.

The second one is probably down to Octopart not returning the price info.

FYI dev-partinfo is currently not making new Octopart requests because we hit the limit. I have requested a higher limit (which will cost $50 a month).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Discution about implementation and new features. feature-request New features resquested.
Projects
None yet
Development

No branches or pull requests

3 participants