Hi Team,Our cluster is currently in DBR 12.1 but it spins up a VMs with Ubuntu 18.04 LTS. 18.04 will be EOL soon. According to this https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.1 OS version should be 20.04 and now a bit...
Hi,I am trying to write a simple code in databricks using langchain. But it is throwing this error: Resource not %pip install --upgrade openai%pip install langchain --upgrade%pip install pymssql --upgrade%pip install SQLAlchemy%pip install pyodbcimpo...
We have a use case of implementing custom logging framework for capturing the various details while executing a notebook. At the end of the notebook, the log file gets copied from driver's /tmp folder into Data lake Storage's container.While this pro...
Hello,I am somewhat new to Databricks and am trying to build a Q&A application based on a collection of documents. I need to move .pdf and .docx files from my local machine to storage in Databricks and eventually a document store. My questions are:Wh...
Hi all,I took an initial stab at task one with some success using the Databricks CLI. Here are the steps below:Open Command/Anaconda prompt and enter: pip install databricks-cliGo to your Databricks console and under settings find "User Settings" and...
I am using DLT to load csv in ADLS, below is my sql query in notebook:CREATE OR REFRESH STREAMING LIVE TABLE test_account_raw AS SELECT * FROM cloud_files( "abfss://my_container@my_storageaccount.dfs.core.windows.net/test_csv/", "csv", map("h...
thank you every one, the problem is resolved, problem is gone when I have workspace admin access.
I am using below code to create and read widgets. I am assigning default value.dbutils.widgets.text("pname", "default","parameter_name")pname=dbutils.widgets.get("pname")I am using this widget parameter in some sql scripts. one example is given below...
I have STRING column in a DLT table that was loaded using SQL Autoloader via a JSON file. When I use the "schema_of_json" function in a SQL statement passing in the literal string from the STRING column then I get this output:ARRAY<STRUCT<firstFetchD...
Databricks recently introduced extension inside the VS code, this is good feature but my company has some security concerns, If I wanted to block connecting to Databricks from Visual studio code, How can I do it? Is there any process which blocks con...
Hi @Kiran Gogula We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful t...
Hi allI have a table created by DLT. Initially I specified cloudFiles.inferColumnTypes to false and all columns are stored as strings. However, I now want to use cloudFiles.inferColumnTypes=true. I dropped the table and re-ran the pipeline, which fai...
Hi @Billy Wong Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
Arrays of complex types seemingly always evaluate to ARRAY<STRING>. Therefore, casting or attempting to load JSON data with empty array values fails. For example, attempting to cast a JSON value of {"likes": []...} on load to the following table sche...
Hi @Jake Neyer Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...
Hi @Andreas Kaae Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
Hey guys,I tried to create a workspace, I didn't encountered error like this. I provided everything correctly but don't know why I'm getting this. Tried doing it by using different Data bricks Id and AWS ID also access this on AWS Root account
https://docs.gcp.databricks.com/administration-guide/workspace/create-workspace.html
Could it be possible, we can retrieve the DBU's on cluster basis within Databricks Notebook itself?This info we get in the compute tab in Databricks for each cluster as Active DBU/hr.
It wont be possible to access DBU used per cluster within DB Clusters.
I know query history api provides filter_by option with start and end time in ms but I was wondering if I can get only the last 5 mins of query data every time I run the api call (using telegraf to call the api). Is it possible I can use relative dat...
I have a table called MetaData and what columns are needed in the select are stored in MetaData.columnsI would like to read columns dynamically from MetaData.columns and create a view based on that.csv_values = "col1, col2, col3, col4"df = spark.crea...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
