- 27198 Views
- 7 replies
- 3 kudos
- 27198 Views
- 7 replies
- 3 kudos
- 3 kudos
By any chance, was the cluster restarted after installing the libraries or was it detached and reattached from/to the notebook? Notebook-scoped libraries do not persist across sessions. You must reinstall notebook-scoped libraries at the beginning of...
- 3 kudos
- 38657 Views
- 5 replies
- 7 kudos
Resolved! How to convert a string column to Array of Struct ?
I have a nested struct , where on of the field is a string , it looks something like this ....string = "[{\"to_loc\":\"6183\",\"to_loc_type\":\"S\",\"qty_allocated\":\"18\"},{\"to_loc\":\"6137\",\"to_loc_type\":\"S\",\"qty_allocated\":\"9\"},{\"to_lo...
- 38657 Views
- 5 replies
- 7 kudos
- 7 kudos
Can you mark the question as answered so others can find the solution?
- 7 kudos
- 4596 Views
- 3 replies
- 0 kudos
- 4596 Views
- 3 replies
- 0 kudos
- 744 Views
- 0 replies
- 0 kudos
Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of i...
Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of income for a large number of people in Kerala. National parks, wild life sanctuaries, etc. are the ma...
- 744 Views
- 0 replies
- 0 kudos
- 2237 Views
- 2 replies
- 0 kudos
Hi all,My organization has changed our domain emails and now all databricks users can't login.We can only login into azure portal with our new dom...
Hi all,My organization has changed our domain emails and now all databricks users can't login.We can only login into azure portal with our new domain email.The message is the following (using the new domain)I wonder if there is a way to upload all us...
- 2237 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Prabakar Ammeappin​ thanks for your response. I wanted to know if the domain name change is transparent within the same workspace. We don't need to migrate data, only replace old domain with new domain. Do you think this is possible?
- 0 kudos
- 13550 Views
- 1 replies
- 1 kudos
Resolved! Maximum duration of the Databricks job before it times out
May I know the duration (max) a job is allowed to run if Timeout is not sethttps://docs.databricks.com/data-engineering/jobs/jobs.html
- 13550 Views
- 1 replies
- 1 kudos
- 1 kudos
This is part of the configuration of the task itself, so if no timeout is specified, it can theoretically run forever (e.g. streaming use case). Please refer timeout section in below link.https://docs.databricks.com/dev-tools/api/latest/jobs.html#ope...
- 1 kudos
- 8900 Views
- 7 replies
- 31 kudos
Resolved! Workspace deployment on AWS - CloudFormation Issue
Hello,I have been trying to deploy a workspace on AWS using the quickstart feature, and I have been running into a problem where the stack fails when trying to create a resource.The following resource(s) failed to create: [CopyZips].From the CloudWat...
- 8900 Views
- 7 replies
- 31 kudos
- 31 kudos
Dropping by with my experience in case anyone lands here via Google.Note that the databricks-prod-public-cfts bucket is located in us-west-2.If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is...
- 31 kudos
- 8370 Views
- 8 replies
- 6 kudos
Resolved! How do you Upload TXT and CSV files into Shared Workspace in Databricks?
I try to upload the needed files under the right directory of the project to work.The files are zipped first as that is an accepted format. I have a Python project which requires the TXT and CSV format files as they are called and used via .py files ...
- 8370 Views
- 8 replies
- 6 kudos
- 6 kudos
@Shay Alam​, can you share the code with which you read the files? Apparently python interprets the file format as a language, so it seems like some options are not filled in correctly.
- 6 kudos
- 4337 Views
- 8 replies
- 1 kudos
Please bring back notebook names in google chrome tabs. This feature seemed to have disappeared within the last 24 hours. Now, each tab just reads &...
Please bring back notebook names in google chrome tabs. This feature seemed to have disappeared within the last 24 hours. Now, each tab just reads "Databricks" at the top. I often have multiple databricks scripts open at the same time and it is re...
- 4337 Views
- 8 replies
- 1 kudos
- 1 kudos
The fix has been pushed to all regions during their release maintenance window. So if your workspace is deployed with the new release, then you should be able to see the notebook names in the browser tab.
- 1 kudos
- 4458 Views
- 2 replies
- 2 kudos
Resolved! Delta Live Tables with CDC and Database Views with Lower Case Names
Hi,I am testing out creating some Delta Live Tables using Change Data Capture and having an issue where the resulting views that are created have lower case column names. Here is my function I am using to ingest data:def raw_to_ods_merge(table_name,s...
- 4458 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Kaniz Fatma​ Not found a solution just yet, but not a priority as most users will be accessing through Databricks SQL, so no further assistance required right now.Thanks
- 2 kudos
- 26406 Views
- 12 replies
- 5 kudos
Displaying Pandas Dataframe
I had this issue when displaying pandas data frames. Any ideas on how to display a pandas dataframe? display(mydataframe) Exception: Cannot call display(<class 'pandas.core.frame.DataFrame'>)
- 26406 Views
- 12 replies
- 5 kudos
- 5 kudos
A simple way to get a nicely formatted table from a pandas dataframe:displayHTML(df.to_html())to_html has some parameters you can control the output with. If you want something less basic, try out this code that I wrote that adds scrolling and some ...
- 5 kudos
- 16367 Views
- 6 replies
- 8 kudos
Resolved! how to flatten non standard Json files in a dataframe
hello, I have a non standard Json file with a nested file structure that I have issues with. Here is an example of the json file. jsonfile= """[ { "success":true, "numRows":2, "data":{ "58251":{ "invoiceno":"58...
- 16367 Views
- 6 replies
- 8 kudos
- 8 kudos
@stale stokkereit​ You can use the below function to flatten the struct fieldimport pyspark.sql.functions as F def flatten_df(nested_df): flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct'] nested_cols = [c[0] for c in nest...
- 8 kudos
- 5361 Views
- 4 replies
- 2 kudos
Resolved! cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch:
Hi,I'm trying to create a delta table using SQL but I'm getting this errorError in SQL statement: AnalysisException: cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch: differing types in '(CAST(10000 AS BIGINT) div Khe)' (big...
- 5361 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Adalberto Garcia Espinosa​ Do you need khe column to be double? If not, below query is working:%sql CREATE OR REPLACE TABLE Productos(Khe bigint NOT NULL,Fctor_HL_Estiba bigint GENERATED ALWAYS AS (cast(10000 as bigint) div Khe)) seems to be work...
- 2 kudos
- 5806 Views
- 5 replies
- 8 kudos
Resolved! Access azure storage account from databricks notebook using pyspark or SQL
I have a storage account - Azure BLOB StorageThere I had container. Inside the container we had a CSV file. Couldn't read the file using the access Key and Storage account name.Any idea how to read file using PySpark/SQL? Thanks in advance
- 5806 Views
- 5 replies
- 8 kudos
- 8 kudos
@Ambiga D​ you need to mount the storage https://docs.databricks.com/data/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-to-dbfs you can follow this,thanks.
- 8 kudos
- 23300 Views
- 5 replies
- 9 kudos
Resolved! How to write *.csv file from DataBricks FileStore
Struggling with how to export a Spark dataframe as a *.csv file to a local computer. I'm successfully using the spark_write_csv funciton (sparklyr R library R) to write the csv file out to my databricks dbfs:FileStore location. Becase (I'm assuming)...
- 23300 Views
- 5 replies
- 9 kudos
- 9 kudos
sparklyr has a different syntax. There is function sdf_coalesce.The code which you paste is for Scala/Python. Additionally, even in python you can only specify folder not file so CSV("dbfs:FileStore/temp/")
- 9 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
