- 4029 Views
- 2 replies
- 1 kudos
Append an empty dataframe to a list of dataframes using for loop in python
I have the following 3 dataframes:I want to append df_forecast to each of df2_CA and df2_USA using a for-loop. However when I run my code, df_forecast is not appending: df2_CA and df2_USA appear exactly as shown above.Here’s the code:df_list=[df2_CA,...
- 4029 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Jack Homareau​ , We haven’t heard from you on the last response from @Arvind Ravish​ , and I was checking back to see if you have a resolution yet. If you have any solution, please do share that with the community as it can be helpful to others. ...
- 1 kudos
- 2760 Views
- 3 replies
- 0 kudos
- 2760 Views
- 3 replies
- 0 kudos
- 388 Views
- 0 replies
- 0 kudos
Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of i...
Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of income for a large number of people in Kerala. National parks, wild life sanctuaries, etc. are the ma...
- 388 Views
- 0 replies
- 0 kudos
- 1226 Views
- 2 replies
- 0 kudos
Hi all,My organization has changed our domain emails and now all databricks users can't login.We can only login into azure portal with our new dom...
Hi all,My organization has changed our domain emails and now all databricks users can't login.We can only login into azure portal with our new domain email.The message is the following (using the new domain)I wonder if there is a way to upload all us...
- 1226 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Prabakar Ammeappin​ thanks for your response. I wanted to know if the domain name change is transparent within the same workspace. We don't need to migrate data, only replace old domain with new domain. Do you think this is possible?
- 0 kudos
- 9361 Views
- 1 replies
- 1 kudos
Resolved! Maximum duration of the Databricks job before it times out
May I know the duration (max) a job is allowed to run if Timeout is not sethttps://docs.databricks.com/data-engineering/jobs/jobs.html
- 9361 Views
- 1 replies
- 1 kudos
- 1 kudos
This is part of the configuration of the task itself, so if no timeout is specified, it can theoretically run forever (e.g. streaming use case). Please refer timeout section in below link.https://docs.databricks.com/dev-tools/api/latest/jobs.html#ope...
- 1 kudos
- 5956 Views
- 9 replies
- 38 kudos
Resolved! Workspace deployment on AWS - CloudFormation Issue
Hello,I have been trying to deploy a workspace on AWS using the quickstart feature, and I have been running into a problem where the stack fails when trying to create a resource.The following resource(s) failed to create: [CopyZips].From the CloudWat...
- 5956 Views
- 9 replies
- 38 kudos
- 38 kudos
Dropping by with my experience in case anyone lands here via Google.Note that the databricks-prod-public-cfts bucket is located in us-west-2.If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is...
- 38 kudos
- 5798 Views
- 8 replies
- 6 kudos
Resolved! How do you Upload TXT and CSV files into Shared Workspace in Databricks?
I try to upload the needed files under the right directory of the project to work.The files are zipped first as that is an accepted format. I have a Python project which requires the TXT and CSV format files as they are called and used via .py files ...
- 5798 Views
- 8 replies
- 6 kudos
- 6 kudos
@Shay Alam​, can you share the code with which you read the files? Apparently python interprets the file format as a language, so it seems like some options are not filled in correctly.
- 6 kudos
- 2899 Views
- 9 replies
- 1 kudos
Please bring back notebook names in google chrome tabs. This feature seemed to have disappeared within the last 24 hours. Now, each tab just reads &...
Please bring back notebook names in google chrome tabs. This feature seemed to have disappeared within the last 24 hours. Now, each tab just reads "Databricks" at the top. I often have multiple databricks scripts open at the same time and it is re...
- 2899 Views
- 9 replies
- 1 kudos
- 1 kudos
The fix has been pushed to all regions during their release maintenance window. So if your workspace is deployed with the new release, then you should be able to see the notebook names in the browser tab.
- 1 kudos
- 20249 Views
- 12 replies
- 4 kudos
Displaying Pandas Dataframe
I had this issue when displaying pandas data frames. Any ideas on how to display a pandas dataframe? display(mydataframe) Exception: Cannot call display(<class 'pandas.core.frame.DataFrame'>)
- 20249 Views
- 12 replies
- 4 kudos
- 4 kudos
A simple way to get a nicely formatted table from a pandas dataframe:displayHTML(df.to_html())to_html has some parameters you can control the output with. If you want something less basic, try out this code that I wrote that adds scrolling and some ...
- 4 kudos
- 9779 Views
- 6 replies
- 7 kudos
Resolved! how to flatten non standard Json files in a dataframe
hello, I have a non standard Json file with a nested file structure that I have issues with. Here is an example of the json file. jsonfile= """[ { "success":true, "numRows":2, "data":{ "58251":{ "invoiceno":"58...
- 9779 Views
- 6 replies
- 7 kudos
- 7 kudos
@stale stokkereit​ You can use the below function to flatten the struct fieldimport pyspark.sql.functions as F def flatten_df(nested_df): flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct'] nested_cols = [c[0] for c in nest...
- 7 kudos
- 3738 Views
- 4 replies
- 2 kudos
Resolved! cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch:
Hi,I'm trying to create a delta table using SQL but I'm getting this errorError in SQL statement: AnalysisException: cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch: differing types in '(CAST(10000 AS BIGINT) div Khe)' (big...
- 3738 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Adalberto Garcia Espinosa​ Do you need khe column to be double? If not, below query is working:%sql CREATE OR REPLACE TABLE Productos(Khe bigint NOT NULL,Fctor_HL_Estiba bigint GENERATED ALWAYS AS (cast(10000 as bigint) div Khe)) seems to be work...
- 2 kudos
- 3905 Views
- 6 replies
- 8 kudos
Resolved! Access azure storage account from databricks notebook using pyspark or SQL
I have a storage account - Azure BLOB StorageThere I had container. Inside the container we had a CSV file. Couldn't read the file using the access Key and Storage account name.Any idea how to read file using PySpark/SQL? Thanks in advance
- 3905 Views
- 6 replies
- 8 kudos
- 8 kudos
@Ambiga D​ you need to mount the storage https://docs.databricks.com/data/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-to-dbfs you can follow this,thanks.
- 8 kudos
- 25697 Views
- 2 replies
- 1 kudos
Resolved! Configuring pip index-url and using artifacts-keyring
Hi I would like to use the azure artifact feed as my default index-url when doing a pip install on a Databricks cluster. I understand I can achieve this by updating the pip.conf file with my artifact feed as the index-url. Does anyone know where i...
- 25697 Views
- 2 replies
- 1 kudos
- 1 kudos
for your first question https://docs.databricks.com/libraries/index.html#python-environment-management and https://docs.databricks.com/libraries/notebooks-python-libraries.html#manage-libraries-with-pip-commands this may help. again you can convert t...
- 1 kudos
- 13512 Views
- 7 replies
- 10 kudos
Resolved! How to write *.csv file from DataBricks FileStore
Struggling with how to export a Spark dataframe as a *.csv file to a local computer. I'm successfully using the spark_write_csv funciton (sparklyr R library R) to write the csv file out to my databricks dbfs:FileStore location. Becase (I'm assuming)...
- 13512 Views
- 7 replies
- 10 kudos
- 10 kudos
Hi @Jeff (Customer), Were you able to follow @Hubert Dudek​ ? Did it help you?
- 10 kudos
- 23239 Views
- 5 replies
- 10 kudos
Resolved! Table write command stuck "Filtering files for query."
Hello all,Background:I am having an issue today with databricks using pyspark-sql and writing a delta table. The dataframe is made by doing an inner join between two tables and that is the table which I am trying to write to a delta table. The table ...
- 23239 Views
- 5 replies
- 10 kudos
- 10 kudos
@Ljuboslav Boskic​ there can be multiple reasons why the query is taking more time , during this phase metadata look-up activity happens, can you please check on the below things Ensuring the tables are z-ordered properly, and that the merge key (on ...
- 10 kudos