I am using below code to create and read widgets. I am assigning default value.dbutils.widgets.text("pname", "default","parameter_name")pname=dbutils.widgets.get("pname")I am using this widget parameter in some sql scripts. one example is given below...
I have STRING column in a DLT table that was loaded using SQL Autoloader via a JSON file. When I use the "schema_of_json" function in a SQL statement passing in the literal string from the STRING column then I get this output:ARRAY<STRUCT<firstFetchD...
Databricks recently introduced extension inside the VS code, this is good feature but my company has some security concerns, If I wanted to block connecting to Databricks from Visual studio code, How can I do it? Is there any process which blocks con...
Hi @Kiran Gogula We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful t...
Hi allI have a table created by DLT. Initially I specified cloudFiles.inferColumnTypes to false and all columns are stored as strings. However, I now want to use cloudFiles.inferColumnTypes=true. I dropped the table and re-ran the pipeline, which fai...
Hi @Billy Wong Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
Arrays of complex types seemingly always evaluate to ARRAY<STRING>. Therefore, casting or attempting to load JSON data with empty array values fails. For example, attempting to cast a JSON value of {"likes": []...} on load to the following table sche...
Hi @Jake Neyer Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...
Hi @Andreas Kaae Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
Hey guys,I tried to create a workspace, I didn't encountered error like this. I provided everything correctly but don't know why I'm getting this. Tried doing it by using different Data bricks Id and AWS ID also access this on AWS Root account
https://docs.gcp.databricks.com/administration-guide/workspace/create-workspace.html
Could it be possible, we can retrieve the DBU's on cluster basis within Databricks Notebook itself?This info we get in the compute tab in Databricks for each cluster as Active DBU/hr.
It wont be possible to access DBU used per cluster within DB Clusters.
I know query history api provides filter_by option with start and end time in ms but I was wondering if I can get only the last 5 mins of query data every time I run the api call (using telegraf to call the api). Is it possible I can use relative dat...
I have a table called MetaData and what columns are needed in the select are stored in MetaData.columnsI would like to read columns dynamically from MetaData.columns and create a view based on that.csv_values = "col1, col2, col3, col4"df = spark.crea...
Hello,I have been working on this issue as a proof of concept - it would be extremely helpful to iterate through tables via loops in a few scenarios. I have a simple three column dimension that I added to a cached table.cache lazy table hedis_cache s...
Got it to work, thank you for the tip! I needed to convert the dataframe over to a pandas dataframehttps://www.geeksforgeeks.org/convert-pyspark-dataframe-to-dictionary-in-python/
I have Updated a Hive Meta Store from a Unity Catalog. I have setup Primary keys on the table. When I try to insert duplicates its succeeding Inserts and seems like PK is not working. Anyone else seeing such behaviour ?
In Databricks, CI/CD process is decoupled or coupled ?
What Serverless features are you using on Databricks? I am curious to know.Is it Databricks SQL Serverless or Model Serving?Proceed here to Compare serverless compute to other Databricks architectureshttps://docs.databricks.com/serverless-compute/ind...
I wanted to add secrets to spark conf of the cluster but i am not able to because i am not the cluster owner. I want to know how can we change the cluster owner?
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
