- 2357 Views
- 1 replies
- 2 kudos
Resolved! Databricks Views
How do we find the definition of View in databricks?
- 2357 Views
- 1 replies
- 2 kudos
- 2 kudos
You can use the extended table description. For example, the following python code will print the current definition of the view: table_name = "" df = spark.sql("describe table extended {}".format(table_name)) df.createOrReplaceTempView("view_desript...
- 2 kudos
- 7079 Views
- 1 replies
- 0 kudos
SQL Alter table command not working for me
I am a novice with databricks. I am performing some independent learning. I am trying to add a column to an existing table. Here is my syntax: %sql ALTER TABLE car_parts ADD COLUMNS (engine_present boolean) which returns the error:SyntaxError: inva...
- 7079 Views
- 1 replies
- 0 kudos
- 0 kudos
Is the table you are working with in the Delta format? The table commands (i.e. Alter) do not work for all storage formats. For example if I run the following commands then I can alter a table. Note - there is no data in the table but the table exist...
- 0 kudos
- 2506 Views
- 1 replies
- 0 kudos
Missing Databricks Datasets
Hi, I am looking at my Databricks workspace and it looks like I am missing DBFS Databricks-dataset root folder. The dbfs root folders I can view are FileStore, local_disk(),mnt, pipelines and user. Can I mount Databricks-dataset or am I missing some...
- 2506 Views
- 1 replies
- 0 kudos
- 0 kudos
If you run the following command do you receive an error? Or do you just get an empty list?dbutils.fs.ls("/databricks-datasets")
- 0 kudos
- 2606 Views
- 1 replies
- 1 kudos
DNS_Analytics Notebook Problems
Hello everyone! First post on the forums, been stuck at this for awhile now and cannot seem to understand why this is happening. Basically, I have been using a seems to be premade Databricks notebook from Databricks themselves for a DNS Analytics exa...
- 2606 Views
- 1 replies
- 1 kudos
- 1 kudos
@NickGoodfella​ , What's the notebook you're looking at, this one? https://databricks.com/notebooks/dns-analytics.html Are you sure all the previous cells executed? this is suggesting there isn't a model at the path that's expected. You can take a lo...
- 1 kudos
- 1865 Views
- 1 replies
- 0 kudos
The State in-stream is growing too large in stream
I have a customer with a streaming pipeline from Kafka to Delta. They are leveraging RocksDB, watermarking for 30min and attempting to dropDuplicates. They are seeing their state grow to 6.2 billion rows--- on a stream that hits at maximum 7000 rows ...
- 1865 Views
- 1 replies
- 0 kudos
- 0 kudos
I've seen a similar issue with large state using flatMapGroupsWithState. It is possible that A.) they are not using the state.setTimeout correctly or B.) they are not calling state.remove() when the stored state has timed out, leaving the state to gr...
- 0 kudos
- 7806 Views
- 2 replies
- 1 kudos
how to calculate median on azure databricks delta table using sql
how to calculate median on delta tables in azure databricks using sql ? select col1, col2, col3, median(col5) from delta table group by col1, col2, col3
- 7806 Views
- 2 replies
- 1 kudos
- 1 kudos
try with the percentile function, as median = percentile 50: https://spark.apache.org/docs/latest/api/sql/#percentile
- 1 kudos
- 3078 Views
- 1 replies
- 0 kudos
Genrated partition column not being used by optimizer
We have created a table using the new generated column feature (https://docs.microsoft.com/en-us/azure/databricks/delta/delta-batch#deltausegeneratedcolumns) CREATE TABLE ingest.MyEvent( data binary, topic string, timestamp timestamp, date dat...
- 3078 Views
- 1 replies
- 0 kudos
- 0 kudos
I think you have to pass a date in your select query instead of a timestamp.The generated column will indeed derive a data from the timestamp and partition by it. But the docs state: When you write to a table with generated columns and you do not ex...
- 0 kudos
- 2333 Views
- 1 replies
- 1 kudos
What could be the issue with parquet file?
when trying to update or display the dataframe, one of the parquet files is having some issue, "Parquet column cannot be converted. Expected: DecimalType(38,18), Found: DOUBLE" What could be the issue?
- 2333 Views
- 1 replies
- 1 kudos
- 1 kudos
try to explicitly cast the double column to dec(38,18) and then do the display.
- 1 kudos
- 2520 Views
- 1 replies
- 0 kudos
"dbfs:" quote added as a prefix to file path
There is a mount path /mnt/folder I am passing filename as a variable from another function and completing the path variable as follows: filename=file.txt path=/mnt/folder/subfolder/+filename When I'm trying to use the path variable is a function, f...
- 2520 Views
- 1 replies
- 0 kudos
- 0 kudos
databricks uses the databricks file system (dbfs) by default. So my guess is you did not mount the path in databricks.
- 0 kudos
- 1915 Views
- 0 replies
- 0 kudos
Kafka consumer and a web server simultaneously, thread blocking problem in microservice
assumptions: There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more clea...
- 1915 Views
- 0 replies
- 0 kudos
- 5161 Views
- 3 replies
- 0 kudos
Double job execution caused by databricks' RemoteServiceExec using databricks-connector
Hello! I'm using databricks-connector to launch spark jobs using python. I've validated that the python version (3.8.10) and runtime version (8.1) are supported by the installed databricks-connect (8.1.10). Everytime a mapPartitions/foreachParti...
- 5161 Views
- 3 replies
- 0 kudos
- 0 kudos
A community forum to discuss working with Databricks Cloud and Spark. ... Double job execution caused by databricks' RemoteServiceExec using databrick.MyBalanceNow
- 0 kudos
- 1724 Views
- 1 replies
- 0 kudos
Bug Report: Date type with year less than 1000 (years 1-999) in spark sql where [solved]
Hi, I noticed unexpected behavior for Date type. If year value is less then 1000 then filtering do not work. Steps:create table test (date Date); insert into test values ('0001-01-01'); select * from test where date = '0001-01-01' Returns 0 rows....
- 1724 Views
- 1 replies
- 0 kudos
- 0 kudos
Hm, seems to work now.
- 0 kudos
- 1830 Views
- 0 replies
- 0 kudos
How to create circles and find diameter in point cloud?
Good day, Everybody. I have a task I should create many circles in a point cloud (my file can be .csv) and calculate their diameter in the area 100m x 100 m. I have the coordinates of the starting point. The circles should be created at the height ...
- 1830 Views
- 0 replies
- 0 kudos
- 1266 Views
- 1 replies
- 0 kudos
We want to do demand forecasting for our supply chain. How should we benefit from Spark in the Usercase development?
We have a series of blogs on the topic which describe the challenges and the best practices on development of demand forecasting usecases on Databricks. Please refer to this blog and the references in it for more info.
- 1266 Views
- 1 replies
- 0 kudos
- 0 kudos
We have a series of blogs on the topic which describe the challenges and the best practices on development of demand forecasting usecases on Databricks. Please refer to this blog and the references in it for more info.
- 0 kudos
- 3081 Views
- 1 replies
- 1 kudos
Resolved! How to keep the original Swedish/Finnish character in the file?
The files are in ANSI format as it shows in the notepad, I could manuelly convert the files to utf8 and read it. But the files are really large. I dont want to download and upload the files. Is there way so i could keep the swedish/finnish characte...
- 3081 Views
- 1 replies
- 1 kudos
- 1 kudos
So the answer was using the option("charset", "iso-8859-1")
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
