- 3008 Views
- 1 replies
- 6 kudos
Resolved! Logic execute when we create an View?
Hi all,I have a small curious about VIEW on Databricks. Could anyone please help me clarify this?Normal database like Postgres or MS SQL, when we define a view, the logic still not execute that time, only run when we query that VIEW.Not sure how VIEW...
- 3008 Views
- 1 replies
- 6 kudos
- 6 kudos
@Nhan Nguyen​ It works the same. CREATE VIEW constructs a virtual table that has no physical data.
- 6 kudos
- 7772 Views
- 2 replies
- 0 kudos
What are the metrics to be considered for monitoring the Databricks
I am very new to Databricks, just setting up with things. I would like to explore various features of Databricks and start playing around with the environment.I am curious to know what are the metrics should be considered for monitoring the complete ...
- 7772 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks is a powerful platform for data engineering, machine learning, and analytics, and it is important to monitor the performance and health of your Databricks environment to ensure that it is running smoothly.Here are a few key metrics that yo...
- 0 kudos
- 3760 Views
- 1 replies
- 5 kudos
org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
- 3760 Views
- 1 replies
- 5 kudos
- 5632 Views
- 3 replies
- 6 kudos
java.lang.NoClassDefFoundError: Could not initialize class org.locationtech.rasterframes.ref.RFRasterSource$
After installing the latest pyrasterframes (v0.10.1) on Azure databricks 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12), I can create a spark session, read the raster data and print the schema. However, when I try to perform any actions on the dat...
- 5632 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Karthick Narendran​ I got the same error as you.I tried every branch in the locationtech repository but failed. Then luckily I found a rasterframe branch for databricks here.https://github.com/mjohns-databricks/rasterframes/tree/0.10.2-databricks...
- 6 kudos
- 5214 Views
- 2 replies
- 2 kudos
SQL Warehouse Configuration Tweaking
I'm new to setting up a DB environment, and have accumulated a couple of questions around configuring a SQL Warehouse1. When creating a SQL warehouse, the smallest size is 2X-Small, which is 4DBU. The pricing calculator (for Azure) implies you can c...
- 5214 Views
- 2 replies
- 2 kudos
- 2 kudos
Docs do show that it uses E8d as you wrote.SQL Warehouses are a different type of compute than All Purpose or Jobs clusters. The SQL warehouses always use Photon. All purpose and Jobs clusters are used for things such as notebooks or Delta Live Tab...
- 2 kudos
- 4422 Views
- 4 replies
- 1 kudos
How to read feature table without target_df / online inference based on filter_condition in databricks feature store
I'm using databricks feature store == 0.6.1. After I register my feature table with `create_feature_table` and write data with `write_Table` I want to read that feature_table based on filter conditions ( may be on time stamp column ) without calling ...
- 4422 Views
- 4 replies
- 1 kudos
- 1 kudos
create_training_set is just a simple Select from delta tables. All feature tables are just registered delta tables. Here is an example code that I used to handle that: customer_features_df = spark.sql("SELECT * FROM recommender_system.customer_fea...
- 1 kudos
- 7098 Views
- 2 replies
- 1 kudos
Resolved! How to use SQL UDFs for Delta Live Table pipelines?
I've been searching for a way to use a SQL UDF for our DLT pipeline. In this case it is to convert a time duration string into INT seconds. How exactly do we use/apply UDFs in this case?
- 7098 Views
- 2 replies
- 1 kudos
- 1 kudos
@GimYou can create Python UDF and then use it in SQL.https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cookbook.html#use-python-udfs-in-sql
- 1 kudos
- 4019 Views
- 0 replies
- 1 kudos
How to pass all dag_run.conf parameters to python_wheel_task
I want to trigger Databricks job from Airflow using DatabricksSubmitRunDeferrableOperator and I need to pass configuration params. Here is excerpt from my code (definition is not complete, only crucial properties):from airflow.providers.databricks.op...
- 4019 Views
- 0 replies
- 1 kudos
- 5343 Views
- 3 replies
- 2 kudos
Resolved! Unable to reproduce Kmeans Clustering results even after setting seed and tolerance
Hi I have been trying to reproduce Kmeans results with no luckHere is my code snippet:from pyspark.ml.clustering import KMeansKMeans(featuresCol=featuresCol, k=clusters, maxIter=40, seed=1, tol = .00001) Can anyone help?
- 5343 Views
- 3 replies
- 2 kudos
- 2 kudos
This issue was due to spark parallelization which doesn't guarantee the same data is assigned to each partition. I was able to resolve this by making sure the same data is assigned to the same partitions :df.repartition(num_partitions, "ur_col_id")d...
- 2 kudos
- 7012 Views
- 3 replies
- 4 kudos
Resolved! Does Databricks offer something like Oracle's dblink?
I am aware, I can load anything into a DataFrame using JDBC, that works well from Oracle sources. Is there an equivalent in Spark SQL, so I can combine datasets as well?Basically something like so - you get the idea...select lt.field1, rt.fie...
- 7012 Views
- 3 replies
- 4 kudos
- 12516 Views
- 5 replies
- 10 kudos
Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely
Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...
- 12516 Views
- 5 replies
- 10 kudos
- 10 kudos
Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.
- 10 kudos
- 13983 Views
- 2 replies
- 3 kudos
Resolved! Trigger on a table
Hello! Is there an equivalent of Create trigger on a table in Databricks sql?CREATE TRIGGER [schema_name.]trigger_nameON table_nameAFTER {[INSERT],[UPDATE],[DELETE]}[NOT FOR REPLICATION]AS{sql_statements}Thank you in advance!
- 13983 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly ...
- 3 kudos
- 1449 Views
- 1 replies
- 0 kudos
Fail to load excel data(timeout) in databricks sample notebook
Im working with the sample notebook named '1_Customer Lifetimes.py' in https://github.com/databricks-industry-solutions/customer-lifetime-valueIn notebook, there is the code like this `%run "./config/Data Extract"`This load excel data however it occu...
- 1449 Views
- 1 replies
- 0 kudos
- 0 kudos
@Seungsu Lee​ It could be a destination host issue, configuration issue or network issue.Hard to guess, first check if your cluster has an access to the public internet by running this command:%sh ping -c 2 google.com
- 0 kudos
- 5053 Views
- 1 replies
- 0 kudos
Parent Hierarchy Queries/ Path Function /Recursive CTE's
Problem Statement:We have a scenario where we get the data from the source in the format of (in actual 20 Levels and number of fields are more than 4 but for ease of understanding let’s consider below)The actual code involved 20 levels of 4-5 fields ...
- 5053 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think that we have anything similar as a built-in function. You'll need to write some custom code to achieve that.
- 0 kudos
- 9698 Views
- 11 replies
- 13 kudos
Resolved! Is it possible to use other databases within Delta Live Tables (DLT)?
I have set up a DLT with "testing" set as the target database. I need to join data that exists in a "keys" table in my "beta" database, but I get an AccessDeniedException, despite having full access to both databases via a normal notebook.A snippet d...
- 9698 Views
- 11 replies
- 13 kudos
- 13 kudos
As an update to this issue: I was running the DLT pipeline on a personal cluster that had an instance profile defined (as per databricks best practises). As a result, the pipeline did not have permission to access other s3 resources (e.g other databa...
- 13 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
