hii tried to pull the report from below query%pythondf = quickbasePull('b5zj8k_pbz5_0_cd5h4wbbp77n4nvp56b4u','bqmnP8jm7',24)but it is giving me error report too largethen i tried below%pythonimport pyqbfrom pyspark.sql import *import pandas as pdqbc ...
I am using SHOW PARTITIONS <<table_name>> to get all the partitions of a table. I want to use max() on the output of this command to get the latest partition for the table.However, I am not able to use SHOW PARTITIONS <<table_name>> in a CTE/sub-quer...
Want to influence the Databricks product roadmap and services? We are looking for feedback from you - our Databricks Community members - to give your feedback and thoughts about your experience with Databricks over the last 6 months in a ~10 minute s...
Still relatively new to Spark and even more so to Delta Live Tables so apologies if I've missed something fundamental but here goes.We are trying to run a notebook via Delta Live Tables, which contains 2 functions decorated by the `dlt.table` decorat...
I have a df where I am calculating values by month. When I run this code on my df it generates the desired results:for i in range(12,len(df.index)): df.iloc[i, 1] = df.iloc[i-12,1]*(((df.iloc[i,3]/100)+(df.iloc[i,6]/100))+1)So far so good. I want...
Hey there @Jack Homareau Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
Hi All,I attended a 2 day ML training during the Data & AI 2022 summit and I received an email from the events team (ataaisummit@typeaevents.com) telling that the recordings for training and related material will be available in my Databricks Academy...
Hi @Sri H ! I am checking on this for you - hang tight! I'll try and get an update asap from the Academy Team.
When i tried to read file from dbfs, it throws error - Caused by: FileReadException: Error while reading file dbfs:/.......................parquet is not a Parquet file. Expected magic number at tail [80, 65, 82, 49] but found [105, 108, 101, 115].Bu...
I have uploaded a sample file using DBFS tab under Data module.But after create table, unable to load that CSV file uploaded data into the table.Getting below error, please let me know the solution.Error in SQL statement: UnsupportedOperationExceptio...
I am using below code to create the Spark session and also loading the csv file. Spark session and loading csv is running well. However SQL query is generating the Parse Exception.%pythonfrom pyspark.sql import SparkSession # Create a SparkSessio...
This is resolved. Below query works fine nowsqldf = spark.sql("select sum(cast(enrollment as float)), sum(cast(growth as float)),`plan type`,`Parent Organization`,state,`Special Needs Plan`,`Plan Name Sec A`, CASE when `Plan ID` between '800' and '89...
Hi Team,As part of "Data Enginering with Databricks" course section "DE 4.2 - Providing Options for External Sources", I can read total number of records of sales_csv table as 10510. The append command in Cmd17 is supposed to increase this number 2x,...
When trying to ingest parquet files with autoloader with the following codedf = (spark .readStream .format("cloudFiles") .option("cloudfiles.format","parquet") .load(filePath))I get the following error:java.lang.UnsupportedOperationException:...
Hi @Ben Bogart This is supported in DBR 11.1 and above.The below document suggests the same:https://docs.databricks.com/ingestion/auto-loader/schema.html#schema-inference-and-evolution-in-auto-loaderPlease try in DBR 11.1 and please let us know if y...
I am writing a streaming job which will be performing ETL for more than 130 tables. I would like to know is there any other better way to do this. Another solution I am thinking is to write separate streaming job for all tables. source data is coming...
Hi, I guess to answer your question it might be helpful to get more details on what you're trying to achieve and the bottleneck that you encounter now.Indeed handle the processing of 130 tables in one monolith could be challenging as the business rul...
I have searched several ways on applying a bold to a text however unable to achieve it.Have added '\033[1m' then my text and followed by '\033[0m', however cant see the text as bold.I need to apply Bold to the Header "Ocean" in below image which is i...
We would like to be able to get the run_id in a job run and we have the unfortunate restriction that we cannot use dbutils, is there a way to get it in python?I know for Job ID it's possible to retrieve it from the environment variables.
Hi, please refer to the following thread : https://community.databricks.com/s/question/0D58Y00008pbkj9SAA/how-to-get-the-job-id-and-run-id-and-save-into-a-databaseHope this helps
i need to convert a spark dataframe to pandas dataframe with arrow optimization spark.conf.set("spark.sql.execution.arrow.enabled", "true")data_df=df.toPandas()but getting one of the below error randomly while doing so Exception: arrow is not support...
Can you confirm this is a known issue?Running into same issue, example to test in 1 cell.# using Arrow fails on HighConcurrency-cluster with PassThrough in runtime 10.4 (and 10.5 and 11.0) spark.conf.set("spark.sql.execution.arrow.pyspark.enabled",...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1610 | |
| 763 | |
| 345 | |
| 286 | |
| 251 |
