- 2750 Views
- 4 replies
- 2 kudos
Hi,I'm trying to create a delta table using SQL but I'm getting this errorError in SQL statement: AnalysisException: cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch: differing types in '(CAST(10000 AS BIGINT) div Khe)' (big...
- 2750 Views
- 4 replies
- 2 kudos
Latest Reply
Hi @Adalberto Garcia Espinosa​ Do you need khe column to be double? If not, below query is working:%sql CREATE OR REPLACE TABLE Productos(Khe bigint NOT NULL,Fctor_HL_Estiba bigint GENERATED ALWAYS AS (cast(10000 as bigint) div Khe)) seems to be work...
3 More Replies
by
Ambi
• New Contributor III
- 2592 Views
- 6 replies
- 8 kudos
I have a storage account - Azure BLOB StorageThere I had container. Inside the container we had a CSV file. Couldn't read the file using the access Key and Storage account name.Any idea how to read file using PySpark/SQL? Thanks in advance
- 2592 Views
- 6 replies
- 8 kudos
Latest Reply
Atanu
Esteemed Contributor
@Ambiga D​ you need to mount the storage https://docs.databricks.com/data/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-to-dbfs you can follow this,thanks.
5 More Replies
- 13960 Views
- 2 replies
- 1 kudos
Hi I would like to use the azure artifact feed as my default index-url when doing a pip install on a Databricks cluster. I understand I can achieve this by updating the pip.conf file with my artifact feed as the index-url. Does anyone know where i...
- 13960 Views
- 2 replies
- 1 kudos
Latest Reply
Atanu
Esteemed Contributor
for your first question https://docs.databricks.com/libraries/index.html#python-environment-management and https://docs.databricks.com/libraries/notebooks-python-libraries.html#manage-libraries-with-pip-commands this may help. again you can convert t...
1 More Replies
by
Jeff1
• Contributor II
- 9342 Views
- 7 replies
- 10 kudos
Struggling with how to export a Spark dataframe as a *.csv file to a local computer. I'm successfully using the spark_write_csv funciton (sparklyr R library R) to write the csv file out to my databricks dbfs:FileStore location. Becase (I'm assuming)...
- 9342 Views
- 7 replies
- 10 kudos
Latest Reply
Hi @Jeff (Customer), Were you able to follow @Hubert Dudek​ ? Did it help you?
6 More Replies
- 13132 Views
- 5 replies
- 10 kudos
- 13132 Views
- 5 replies
- 10 kudos
Latest Reply
@Ljuboslav Boskic​ there can be multiple reasons why the query is taking more time , during this phase metadata look-up activity happens, can you please check on the below things Ensuring the tables are z-ordered properly, and that the merge key (on ...
4 More Replies
- 418 Views
- 0 replies
- 0 kudos
What is a natural resource and why do we need them?Natural resources are vital to our survival and well-being. They provide the food, water, and energy that we need to live, and they support the ecosystems that we rely on for our livelihoods.However,...
- 418 Views
- 0 replies
- 0 kudos
- 328 Views
- 0 replies
- 2 kudos
June Featured Member of the Month ! Werner Stinckens Job Title: Data Engineer @ Van de Velde (www.vandevelde.eu)What are three words your coworkers would use to describe you?Helpful, accurate, inquisitiveWhat is your favorite thing about your curren...
- 328 Views
- 0 replies
- 2 kudos
- 4709 Views
- 13 replies
- 2 kudos
Hi, I am very new in databricks and I am trying to run quick experiments to understand the best practice for me, my colleagues and the company.I pull the data from snowflakedf = spark.read \ .format("snowflake") \ .options(**options) \ .option('qu...
- 4709 Views
- 13 replies
- 2 kudos
- 6139 Views
- 5 replies
- 12 kudos
Hi,How to convert each row of dataframe to array of rows?Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().colle...
- 6139 Views
- 5 replies
- 12 kudos
Latest Reply
@Hubert Dudek​ , Thank you for the reply. We are new to ADB. And using the below code, looking for an optimized way to do itdfJSONString = df.toJSON().collect()stringList = [] for row in dfJSONString: # ==== Unflatten the JSON string ==== # js...
4 More Replies
by
Alix
• New Contributor III
- 7304 Views
- 9 replies
- 3 kudos
Hello,I've been trying to submit a job to a transient cluster, but it is failing with this error :Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in ...
- 7304 Views
- 9 replies
- 3 kudos
Latest Reply
Hi @Alix Métivier​ , Just a friendly follow-up. Do you still need help, or @Shanmugavel Chandrakasu​ 's response help you to find the solution? Please let us know.
8 More Replies
- 25502 Views
- 11 replies
- 1 kudos
from pyspark import SparkContextfrom pyspark import SparkConffrom pyspark.sql.types import *from pyspark.sql.functions import *from pyspark.sql import *from pyspark.sql.types import StringTypefrom pyspark.sql.functions import udfdf1 = spark.read.form...
- 25502 Views
- 11 replies
- 1 kudos
Latest Reply
Thanks i modified my code as per your suggestion and it worked perfectly Thanks again for all your inputsdflist= spark.createDataFrame(list(a.columns), "string").toDF("Name")dfg=dflist.filter(col('name').isin('ref_date')).count()if dfg==1 : a = a.wi...
10 More Replies
- 296 Views
- 0 replies
- 0 kudos
What are Canyon Creek Condos and what do they offer residents?Canyon Creek Condos are a type of housing that is becoming increasingly popular in the United States. These types of condos are typically located in rural or suburban areas and offer resid...
- 296 Views
- 0 replies
- 0 kudos
- 1581 Views
- 5 replies
- 2 kudos
Hello all, I am trying to use the Map visualization in SQL Analytics Dashboard in Databricks. Does any one knows how or if we can change the size/radius of the markers based on values in another column. This seems like a very trivial parameter but I ...
- 1581 Views
- 5 replies
- 2 kudos
- 2352 Views
- 5 replies
- 1 kudos
Hi All, We are trying to use the Spark 3 structured streaming feature/option ".option('cleanSource','archive')" to archive processed files. This is working as expected using the standard spark implementation, however does not appear to work using aut...
- 2352 Views
- 5 replies
- 1 kudos
Latest Reply
https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-optionscleanSource is not a listed option so it won't do anything.Maybe event retention is something you can use?
4 More Replies
- 4067 Views
- 6 replies
- 6 kudos
I'm reshaping my dataframe as per requirement and I came across this situation where I'm concatenating 2 dataframes and then transposing them. I've done this previously using pandas and the syntax for pandas goes as below:import pandas as pd
df1 = ...
- 4067 Views
- 6 replies
- 6 kudos
Latest Reply
Hi @Kaniz Fatma​ ,I no longer see the answer you've posted, but I see you were suggesting to use `union`. As per my understanding, union are used to stack the dfs one upon another with similar schema / column names.In my situation, I have 2 different...
5 More Replies
