- 319 Views
- 1 replies
- 1 kudos
Resolved! How to Find DBU Consumption and Cost for a Serverless Job?
Hello community,I'm new to using Serverless compute for my Jobs and I need some help understanding how to monitor the costs.I have configured and run a job that executes a notebook using the "Serverless" compute option. The job completed successfully...
- 319 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @LonguiVic1 ,You can use system table to track consumption in serverless. In below article they even provide sample queries you can use. Also, notice that there's list_prices system table that includes list prices over time for each available SKU....
- 1 kudos
- 134 Views
- 1 replies
- 0 kudos
Databricks asset bundles in Python: referencing variables
Hey,I am using DAB's and in my .yml files I can reference my variables set in my databricks.yml like this: git_branch: ${var.branch}I would like to do the same thing in my DAB's written in Python but I cannot find any documentation on how to do this....
- 134 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @daan_dw ,To reference variables defined in your databricks.yml in Python DAB code, define your variables class and use bundle.resolve_variablehttps://docs.databricks.com/aws/en/dev-tools/bundles/python/#access-bundle-variables
- 0 kudos
- 99 Views
- 0 replies
- 0 kudos
DQX - datacontract cli
Hello Guyz, Someone can i combine dqx databricks rules check with datacontract cli ? If yes can we share your idea? https://gpt.datacontract.com/sources/cli.datacontract.com/Cordially,
- 99 Views
- 0 replies
- 0 kudos
- 251 Views
- 1 replies
- 1 kudos
Resolved! Online Table Migration
I am currently trying to migrate our Online Tables to synced tables with Online Feature Store since Online Tables is deprecated. When creating a new table, it worked just fine and how the docs said it would (https://docs.databricks.com/aws/en/machine...
- 251 Views
- 1 replies
- 1 kudos
- 1 kudos
Migrating from deprecated Online Tables to synced tables with the Databricks Online Feature Store can be tricky due to several points of integration and timing between Unity Catalog (UC), Feature Store metadata, and the underlying online store. The m...
- 1 kudos
- 2342 Views
- 1 replies
- 0 kudos
Resolved! Spark JDBC Netsuite error - SQLSyntaxErrorException: [NetSuite][OpenAccess SDK JDBC Driver][OpenAcc
I'm trying to query the Customer netsuite tables with spark jdbc and I've added the .jar file to the cluster and trying to run the below:jdbc_url = "jdbc:ns://xxxx.connect.api.netsuite.com:1708;ServerDataSource=NetSuite2.com;Encrypted=1;NegotiateS...
- 2342 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are on a Spark version that supports .option("query", ...), you can do: python df = spark.read \ .format("jdbc") \ .option("url", jdbc_url) \ .option("query", "SELECT TOP 10 * FROM Customer") \ .option("user", "xxxx") \ ...
- 0 kudos
- 217 Views
- 2 replies
- 1 kudos
Declarative Pipeline Failure for Autoloader
Hello Folks, After moving my working serverless Auto Loader notebook to a declarative (DLT) pipeline, I’m getting an AccessDenied error. What could be causing this?”here is the DLT json and error message in the DLT I googled around got saw some hint...
- 217 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @iskidet! Were you able to resolve the AccessDenied issue? If the above suggestion helped, or if you found another solution, it would be great if you could mark it as the accepted solution or share your approach with the community.
- 1 kudos
- 462 Views
- 7 replies
- 4 kudos
Resolved! [CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)
Hello,after databricks update the Runtime from Release: 15.4.24 to Release: 15.4.25 we getting in all jobs the Error:[CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)What we can do here?Greetings
- 462 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Hansjoerg,Apologies for the confusion earlier. You are right Bundles doesn't allow pinning to specific patch versions like 15.4.24.Your best option is to skip Bundles for now and use the regular Databricks Jobs setup (via UI or Jobs API) where yo...
- 4 kudos
- 2244 Views
- 1 replies
- 1 kudos
Resolved! Syncing lakebase table to delta table
I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the ta...
- 2244 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey AkhileshVB! Lakebase-to-Delta CDC sync is in Private Preview—GA/Preview dates are not firm yet. Doo you have a Databricks contact or account manager? They are the right place to go if you want early involvement. Workarounds & DIY Approaches: Fo...
- 1 kudos
- 319 Views
- 5 replies
- 2 kudos
Read files from adls in databricks
I have unity catalogue access connector but its not enabled as i have only admin access so i dont have access to the admin portal to enable this as its need global admin permissions.I am trying to read adls json data in databricks by using service pr...
- 319 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @SuMiT1 once networking issue is resolved , also make sure your service principal has at least Storage Blob Data Reader on the storage account/container.
- 2 kudos
- 403 Views
- 3 replies
- 2 kudos
Convert notebook to Python library
Looking for ways to convert a Databricks notebook to Python library. Some context :Don't want to give execute permissions to shared notebooks as we want to hide code from users.Proposed solution is to have our shared notebook converted into a Python ...
- 403 Views
- 3 replies
- 2 kudos
- 2 kudos
The best way to share code from a Databricks notebook as a reusable module while hiding implementation details from users—without using wheels or granting direct notebook execution permissions—is to convert your notebook into a Python module, store i...
- 2 kudos
- 298 Views
- 2 replies
- 2 kudos
Best Practice: Data Modeling for Customer 360 with Refined/Gold Source Data
Hi community,I'm looking for advice on the best data modeling approach for a Customer 360 (C360) project where our source data is already highly refined.I understand the standard Medallion architecture guidelines, which often recommend using Data Vau...
- 298 Views
- 2 replies
- 2 kudos
- 2 kudos
@BS_THE_ANALYST Thank you so much for your response.The goal is to keep it flexible as a platform rather than a data product mindset. Keeping this in mind, essentially the customer data platform should enable contribution from different teams prevent...
- 2 kudos
- 281 Views
- 3 replies
- 0 kudos
SQL Editor Auto complete
HiFrom time to time the SQL Editor Auto complete works and from time to time not.few times it knows the table columns and few time not - what is the rule for it?
- 281 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @pinikrisher ,To be honest I didn't notice this behaviour. Are you using SQL Editor v2 or legacy one?
- 0 kudos
- 575 Views
- 4 replies
- 4 kudos
How to Read Shared Drive Data in Databricks
Hi everyone,I am working on a project where the data is stored on a Shared Drive. How can I read an Excel file from the Shared Drive into a Databricks notebook?Thanks,
- 575 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Akshay_Petkar ,Could you provide more information. Share drive is pretty broad term. It could be Windows SMB / CIFS share , AWS FSx,, Google Shared Drive etc.
- 4 kudos
- 4406 Views
- 5 replies
- 3 kudos
Set time out or Auto termination for long running query
Hi ,We want to set auto termination for long running queries in data bricks adhoc cluster.I attempted below two approaches in my notebook. Despite my understanding that queries should automatically terminate after one hour, with both the approaches q...
- 4406 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @NehaR are the configurations(spark.sql.broadcastTimeout or spark.sql.execution.timeout) working when we set at cluster level. I am currently trying to do a similar configuration for compute clusters but it dosent work.
- 3 kudos
- 207 Views
- 1 replies
- 0 kudos
split parse_url output for the information
Hi All,I have data in blobs which I am loading from blob store to Databricks delta tables. One of the blob types contains urls. From the Urls I want to extract knowledge from the path and query parts I can get those out easily with parse url. the pro...
- 207 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @turagittech ,Honestly, it all depends on how complex your URLs can get.UDFs will always be more flexible but less performant than native SQL functions.That said, if your team mainly works with SQL, trying to solve it natively in Databricks SQL...
- 0 kudos
| User | Count |
|---|---|
| 1617 | |
| 790 | |
| 471 | |
| 349 | |
| 287 |
