- 12070 Views
- 2 replies
- 3 kudos
Nested struct type not supported pyspark error
I am attempting to apply a function to a pyspark DataFrame and save the API response to a new column and then parse using `json_normalize`. This works fine in pandas, however, I run into an exception with `pyspark`. import pyspark.pandas as ps i...
- 12070 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Keval Shah Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 3 kudos
- 7330 Views
- 2 replies
- 2 kudos
Running databricks job with different parameter automatically
I have a python script running as databricks job. Is there a way I can run this job with different set of parameters automatically or programmatically without using run with different parameter option available in UI ?
- 7330 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Divya Bhadauria We haven't heard from you since the last response from @Lakshay Goel , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...
- 2 kudos
- 3180 Views
- 0 replies
- 0 kudos
Snowpark vs Spark on Databricks
Why / When should we choose Spark on Databricks over Snowpark if the data we are processing is underlying in Snowflake?
- 3180 Views
- 0 replies
- 0 kudos
- 2214 Views
- 0 replies
- 0 kudos
Databricks Runtime 12.1 spins VM in Ubuntu 18.04 LTS
Hi Team,Our cluster is currently in DBR 12.1 but it spins up a VMs with Ubuntu 18.04 LTS. 18.04 will be EOL soon. According to this https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.1 OS version should be 20.04 and now a bit...
- 2214 Views
- 0 replies
- 0 kudos
- 2857 Views
- 0 replies
- 0 kudos
Resource Not Found
Hi,I am trying to write a simple code in databricks using langchain. But it is throwing this error: Resource not %pip install --upgrade openai%pip install langchain --upgrade%pip install pymssql --upgrade%pip install SQLAlchemy%pip install pyodbcimpo...
- 2857 Views
- 0 replies
- 0 kudos
- 1225 Views
- 0 replies
- 1 kudos
Custom application log files copy fails in Shared Unity Catalog Cluster
We have a use case of implementing custom logging framework for capturing the various details while executing a notebook. At the end of the notebook, the log file gets copied from driver's /tmp folder into Data lake Storage's container.While this pro...
- 1225 Views
- 0 replies
- 1 kudos
- 6882 Views
- 1 replies
- 2 kudos
Resolved! Building a Document Store on Databricks
Hello,I am somewhat new to Databricks and am trying to build a Q&A application based on a collection of documents. I need to move .pdf and .docx files from my local machine to storage in Databricks and eventually a document store. My questions are:Wh...
- 6882 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi all,I took an initial stab at task one with some success using the Databricks CLI. Here are the steps below:Open Command/Anaconda prompt and enter: pip install databricks-cliGo to your Databricks console and under settings find "User Settings" and...
- 2 kudos
- 2967 Views
- 3 replies
- 1 kudos
Issues to load from ADLS in DLT
I am using DLT to load csv in ADLS, below is my sql query in notebook:CREATE OR REFRESH STREAMING LIVE TABLE test_account_raw AS SELECT * FROM cloud_files( "abfss://my_container@my_storageaccount.dfs.core.windows.net/test_csv/", "csv", map("h...
- 2967 Views
- 3 replies
- 1 kudos
- 1 kudos
thank you every one, the problem is resolved, problem is gone when I have workspace admin access.
- 1 kudos
- 12095 Views
- 2 replies
- 2 kudos
Facing issue while using widget values in sql script
I am using below code to create and read widgets. I am assigning default value.dbutils.widgets.text("pname", "default","parameter_name")pname=dbutils.widgets.get("pname")I am using this widget parameter in some sql scripts. one example is given below...
- 12095 Views
- 2 replies
- 2 kudos
- 3586 Views
- 0 replies
- 0 kudos
Trying to convert STRING column into Array of Structs in SQL statement
I have STRING column in a DLT table that was loaded using SQL Autoloader via a JSON file. When I use the "schema_of_json" function in a SQL statement passing in the literal string from the STRING column then I get this output:ARRAY<STRUCT<firstFetchD...
- 3586 Views
- 0 replies
- 0 kudos
- 2448 Views
- 2 replies
- 0 kudos
Databricks extension in Visual studio Code
Databricks recently introduced extension inside the VS code, this is good feature but my company has some security concerns, If I wanted to block connecting to Databricks from Visual studio code, How can I do it? Is there any process which blocks con...
- 2448 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Kiran Gogula We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful t...
- 0 kudos
- 5897 Views
- 2 replies
- 1 kudos
Overwriting schema in Delta Live Tables
Hi allI have a table created by DLT. Initially I specified cloudFiles.inferColumnTypes to false and all columns are stored as strings. However, I now want to use cloudFiles.inferColumnTypes=true. I dropped the table and re-ran the pipeline, which fai...
- 5897 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Billy Wong Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 5735 Views
- 1 replies
- 1 kudos
Arrays of complex type always evaluate to ARRAY<STRING>?
Arrays of complex types seemingly always evaluate to ARRAY<STRING>. Therefore, casting or attempting to load JSON data with empty array values fails. For example, attempting to cast a JSON value of {"likes": []...} on load to the following table sche...
- 5735 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Jake Neyer Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 4888 Views
- 3 replies
- 5 kudos
Resolved! Training models on big or small clusters
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...
- 4888 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Andreas Kaae Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 5 kudos
- 7181 Views
- 2 replies
- 4 kudos
Resolved! Workspace not getting created
Hey guys,I tried to create a workspace, I didn't encountered error like this. I provided everything correctly but don't know why I'm getting this. Tried doing it by using different Data bricks Id and AWS ID also access this on AWS Root account
- 7181 Views
- 2 replies
- 4 kudos
- 4 kudos
https://docs.gcp.databricks.com/administration-guide/workspace/create-workspace.html
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
