- 3796 Views
- 2 replies
- 1 kudos
Databricks Academy - Access to training recording attended during Data & AI Summit 2022
Hi All,I attended a 2 day ML training during the Data & AI 2022 summit and I received an email from the events team (ataaisummit@typeaevents.com) telling that the recordings for training and related material will be available in my Databricks Academy...
- 3796 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Sri H​ ! I am checking on this for you - hang tight! I'll try and get an update asap from the Academy Team.
- 1 kudos
- 8953 Views
- 3 replies
- 3 kudos
Resolved! Unable to read file from dbfs location in databricks.
When i tried to read file from dbfs, it throws error - Caused by: FileReadException: Error while reading file dbfs:/.......................parquet is not a Parquet file. Expected magic number at tail [80, 65, 82, 49] but found [105, 108, 101, 115].Bu...
- 8953 Views
- 3 replies
- 3 kudos
- 2063 Views
- 0 replies
- 0 kudos
Unable to load table in Community Edition using COPY INTO through DBFS
I have uploaded a sample file using DBFS tab under Data module.But after create table, unable to load that CSV file uploaded data into the table.Getting below error, please let me know the solution.Error in SQL statement: UnsupportedOperationExceptio...
- 2063 Views
- 0 replies
- 0 kudos
- 22367 Views
- 3 replies
- 0 kudos
Resolved! I am getting ParseException: error while running the spark SQL query
I am using below code to create the Spark session and also loading the csv file. Spark session and loading csv is running well. However SQL query is generating the Parse Exception.%pythonfrom pyspark.sql import SparkSession # Create a SparkSessio...
- 22367 Views
- 3 replies
- 0 kudos
- 0 kudos
This is resolved. Below query works fine nowsqldf = spark.sql("select sum(cast(enrollment as float)), sum(cast(growth as float)),`plan type`,`Parent Organization`,state,`Special Needs Plan`,`Plan Name Sec A`, CASE when `Plan ID` between '800' and '89...
- 0 kudos
- 1327 Views
- 0 replies
- 0 kudos
The REFRESH TABLE command doesn't seem to invalidate the local cache. Am I missing something?
Hi Team,As part of "Data Enginering with Databricks" course section "DE 4.2 - Providing Options for External Sources", I can read total number of records of sales_csv table as 10510. The append command in Cmd17 is supposed to increase this number 2x,...
- 1327 Views
- 0 replies
- 0 kudos
- 4239 Views
- 3 replies
- 0 kudos
When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?
When trying to ingest parquet files with autoloader with the following codedf = (spark .readStream .format("cloudFiles") .option("cloudfiles.format","parquet") .load(filePath))I get the following error:java.lang.UnsupportedOperationException:...
- 4239 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ben Bogart​ This is supported in DBR 11.1 and above.The below document suggests the same:https://docs.databricks.com/ingestion/auto-loader/schema.html#schema-inference-and-evolution-in-auto-loaderPlease try in DBR 11.1 and please let us know if y...
- 0 kudos
- 2965 Views
- 2 replies
- 2 kudos
How to handle 100+ tables ETL through spark structured streaming?
I am writing a streaming job which will be performing ETL for more than 130 tables. I would like to know is there any other better way to do this. Another solution I am thinking is to write separate streaming job for all tables. source data is coming...
- 2965 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, I guess to answer your question it might be helpful to get more details on what you're trying to achieve and the bottleneck that you encounter now.Indeed handle the processing of 130 tables in one monolith could be challenging as the business rul...
- 2 kudos
- 11792 Views
- 3 replies
- 4 kudos
Resolved! How to bold a text ?
I have searched several ways on applying a bold to a text however unable to achieve it.Have added '\033[1m' then my text and followed by '\033[0m', however cant see the text as bold.I need to apply Bold to the Header "Ocean" in below image which is i...
- 11792 Views
- 3 replies
- 4 kudos
- 4130 Views
- 2 replies
- 1 kudos
Resolved! How to obtain run_id without using dbutils in python?
We would like to be able to get the run_id in a job run and we have the unfortunate restriction that we cannot use dbutils, is there a way to get it in python?I know for Job ID it's possible to retrieve it from the environment variables.
- 4130 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, please refer to the following thread : https://community.databricks.com/s/question/0D58Y00008pbkj9SAA/how-to-get-the-job-id-and-run-id-and-save-into-a-databaseHope this helps
- 1 kudos
- 10217 Views
- 4 replies
- 5 kudos
Resolved! High Concurrency Pass Through Cluster : pyarrow optimization not working while converting to pandasdf
i need to convert a spark dataframe to pandas dataframe with arrow optimization spark.conf.set("spark.sql.execution.arrow.enabled", "true")data_df=df.toPandas()but getting one of the below error randomly while doing so Exception: arrow is not support...
- 10217 Views
- 4 replies
- 5 kudos
- 5 kudos
Can you confirm this is a known issue?Running into same issue, example to test in 1 cell.# using Arrow fails on HighConcurrency-cluster with PassThrough in runtime 10.4 (and 10.5 and 11.0) spark.conf.set("spark.sql.execution.arrow.pyspark.enabled",...
- 5 kudos
- 3788 Views
- 3 replies
- 1 kudos
Resolved! Databricks SQL duplicates alert e-mail
Hi everyone, I've been working in DBC SQL and creating few e-mails alerts and have noticed that when it triggers, e-mail notification is getting duplicated. I've been trying lots of testing in different situations, however, it keeps duplicating in my...
- 3788 Views
- 3 replies
- 1 kudos
- 1 kudos
After lots of testing, I've finally figured out one solution. I've changed "Notifications" settings to "When triggered, send notification At most every 1 day" and "Refresh" to "Refresh every 1 day". Now it is working perfectly.
- 1 kudos
- 1612 Views
- 1 replies
- 2 kudos
Any suggestions around automating sizing of clusters and best practices around it? Other than enabling auto scaling, are there any other practices aro...
Any suggestions around automating sizing of clusters and best practices around it? Other than enabling auto scaling, are there any other practices around creating a right size driver/worker nodes?
- 1612 Views
- 1 replies
- 2 kudos
- 2 kudos
Autoscaling should help in sizing the clusters according to the workload. You may want to consider the recommendations here: https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-sizing-considerations
- 2 kudos
- 3306 Views
- 3 replies
- 0 kudos
Resolved! docs.databricks.com
Is there an any example that contains at least one widget that works with the Databricks SQL Create Dashboard API? I tried the following simple dashboard:{ "name": "Delta View", "dashboard_filters_enabled": false, "widgets": [ { ...
- 3306 Views
- 3 replies
- 0 kudos
- 0 kudos
@Valentin Rosca​ , Right now, Databricks also does not recommend creating new widgets via queries and dashboards API (https://docs.databricks.com/sql/api/queries-dashboards.html#operation/sql-analytics-create-dashboard). Also, copying a dashboard fr...
- 0 kudos
- 3025 Views
- 2 replies
- 3 kudos
- 3025 Views
- 2 replies
- 3 kudos
- 3 kudos
All good. I just imported the training material manually using the dbc link. Didn't need repos for that.
- 3 kudos
- 3555 Views
- 2 replies
- 1 kudos
I have a big problem creating a community account
It is impossible for me create a community account. I put my data on web and in the next step, when the website show me the 3 type of data ( google, amazn etc) and I click on the “ "Get started with community account" the web show me this I have try...
- 3555 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @david vazquez​,It seems like the website was down due to maintenance. You can check the status page next time to check why the website is down https://status.databricks.com/
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
